Відео

This was great, much easier on my Windows machine than my Linux machine for a change. Just a question, your table definitions in the video for AdventureWorks don't appear to be the same as the available ones on the Microsoft site for versions 2019 or 2022. I created some dummy tables based on the same table definitions in your GitHub, but obviously my dummy data doesn't relate, so I can't properly test if this model is properly generating the correct SQL. Do you have a link to the database you were using?

Hi, Is the connector name and topic name always same? Can you name your ropic something else? If you want to have multiple topic for 1 connector then it will be helpful. Thanks in advance.

thanks a lot for the detailed video, i want to ask about he accuracy of results? is it accurate or not for big datasets

Teacher, where is the source of this data? I would like to insert them into my database. In my case I will insert it into PostgreSQL, run the ETL and write it to s3. Could you provide me with the source?

can we run llama 3 locally on any simple VPS Server, or do we need GPUS ?

Hi there do you have dockeryml file containing mssql connection

Great video What if the response from database exhaustes the context window of the model.

Wonderful as always and just in time. Was going to build a similar use case that auto generates database docs for business users next week. This comes in handy🎉 Thank you again and again

Hi. I encountered this error when trying to add debezium connector via api call. Would you please help? Thanks. Failed testing connection for jdbc:postgresql://localhost:5432/AdventureWorks with user 'etl': Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. [io.debezium.connector.postgresql.PostgresConnector]

What happens if you kill the airflow web server, or localhost? Will the DAG still run on the schedule you specified?

Namaste Haq !!! Thank you so much for making this video!, and also sharing your repo, I'm bit confused how you build the connection string. would you mind to share it? UI had checked you Connect to SQL Server with Python notebook also, but didn't realize what's is not correct on my ConnectionStringCredentials()!

❤ Thank you for this amazing video!

Great video 👏🏻

Hi, would you suggest this method to extract data from an active postgres table that has ~5billion rows? If not do you have any videos on what method you would suggest to extract from postgres to s3? Thanks! (Tried with airbyte but keeps failing)

Great job, keep going 👍

i lover uoui

def test_Genre_dtype_str(df): assert (df["Genre"].dtype == str or df["Genre"].dtype == 'O') This test case is always returned Pass

Thanks for sharing this, I was trying to set this whole thing on local, I am able to run queries using Trino CLI inside docker but I am not able to connect DBeaver to Trino, can you share DBeaver connection configuration??

Love how Kafka is turning into a datalake now that you can have unlimited retention at very cheap cost (KIP-405). This means that you can reduce data movement by bringing analysts directly where the data was ingested. This opens a plethora of new data sources and much greater volume of data available for ad hoc analysis. We can finally say bye to the cost, complexity and consistency issues associated with heavy ELT/ETL processes!

Hi! Thank you for this great video! I'm still not very clear about why we need hive if we already use iceberg - in iceberg, we also can get a schema from minio raw data directly and we twice create the same table in hive and iceberg, but we will use iceberg most of the time. Did I miss something?

Great video 👏🏻

Thanks for the video. I installed pip install jupyter_scheduler but create notebook job option is not listed. Could you please help

Help me please, I can't download the output files when i clicked download icon, Nothing happened at all 😢

Thanks for the informative content! so how would you deal with large tables especially for the initial loads? assume that table is 200-300GB, selecting all data and keeping in data frames/ in memory objects doesn't look practical so I believe defining a batch key/partitions on source side and iterate it on codes could be a way..

This may be helpful to anyone who is facing the issue I go it myself so excited to share. If you are getting this error "Data load error: (psycopg2.errors.InsufficientPrivilege) permission denied for schema public LINE 2: CREATE TABLE "stg_DimProduct" (" First install pip install psycopg2-binary and then I suggest to give permission to etl user. The creator of the video has provided the github repo so he has actually provided the script. But for your convenience you can run this code in your postgresql query tool in pgadmin "CREATE SCHEMA IF NOT EXISTS etl AUTHORIZATION postgres; GRANT ALL ON SCHEMA etl TO etl; GRANT ALL ON SCHEMA etl TO postgres;" and then run the code the data will be loaded/imported to the postgresql database....

one of the best tuto's on youtube, thank you so much !

Very informative

Thanks for this awesome man

i subscribe , send my phone to Bangladesh, Uttara, Sector#13, Road#18. I prefer macbook

Thank you for the vídeo, excelent explanation.

Thank you, i have a question If the task is scheduled to run daily and new data has been inserted into the source since the last transfer, will just new data get transferred next task or all data again

Awesome tutorial.

Great explanation. What library is great for styling? Xlswriter or openpyxl?

I heard that in order to access on premises MS SQL database , the server must be accessible through the internet is that always true. As per your example you have mentioned on premises if that is the case what are the settings in Airflow to connect to ms sql server. what is need to install or what should be in the Docker file or package or requirements file in the project folder .

Thanks for making it so simple...the best so far for beginners i have seen

Hello,when i hit docker-compose up, i get no configuration file provided:not found, and when i tried to transfer another yaml file from different source in github to myfolder i get invalid spec: workspace:: empty section between colons, and i don't know how to solve the problem

great project, congrats. keep going on

Hi! Thank you for the video - it's great explanation as always on your channel! I have a questions. I have similar task starting from kafka and now I'm using iceberg/dremio/nessie stack for storing the data from your previous video. Here you have added hive - could you explain what's benefits of using hive with or instead of stack from your previous data lakehouse guide. Thanks!

🔥🎆😍

Link to Kafka series: ua-cam.com/play/PLaz3Ms051BAkwR7d9voHsflTRmumfkGVW.html Link to Data Lake video: ua-cam.com/video/DLRiUs1EvhM/v-deo.html Link to data lake GitHub repo: github.com/hnawaz007/pythondataanalysis/tree/main/data-lake Link to Kafka GitHub repo: github.com/hnawaz007/pythondataanalysis/tree/main/kafka

This is what happened after I did pip install virtualenv: Collecting virtualenv Downloading virtualenv-20.26.2-py3-none-any.whl.metadata (4.4 kB) Collecting distlib<1,>=0.3.7 (from virtualenv) Downloading distlib-0.3.8-py2.py3-none-any.whl.metadata (5.1 kB) Collecting filelock<4,>=3.12.2 (from virtualenv) Downloading filelock-3.14.0-py3-none-any.whl.metadata (2.8 kB) Collecting platformdirs<5,>=3.9.1 (from virtualenv) Downloading platformdirs-4.2.2-py3-none-any.whl.metadata (11 kB) Downloading virtualenv-20.26.2-py3-none-any.whl (3.9 MB) ---------------------------------------- 3.9/3.9 MB 8.9 MB/s eta 0:00:00 Downloading distlib-0.3.8-py2.py3-none-any.whl (468 kB) ---------------------------------------- 468.9/468.9 kB 2.7 MB/s eta 0:00:00 Downloading filelock-3.14.0-py3-none-any.whl (12 kB) Downloading platformdirs-4.2.2-py3-none-any.whl (18 kB) Installing collected packages: distlib, platformdirs, filelock, virtualenv Successfully installed distlib-0.3.8 filelock-3.14.0 platformdirs-4.2.2 virtualenv-20.26.2

Link to Ollama set up vidoe: ua-cam.com/video/CE9umy2NlhE/v-deo.html Link to complete AI series: hnawaz007.github.io/ai.html

please how do i access the file

wow these videos makes dbt a very easy tool to use

why not put part numbers as part of title?

👏👏👏👏