Skip to main content

Sqoop


For Sqoop with MySQL, It requires MySQL connector jar in Sqoop binary location.

Sqoop binary location: usr/hdp/current/sqoop

Sqoop Commands:

#List Databases
> sqoop list-databases --connect jdbc:mysql://<mysql-URL>:<port> --username <USERNAME> --password <PASSWORD>

#List Table from database
> sqoop list-tables --connect jdbc:mysql://<mysql-URL>:<port>/<DBName> --username <USERNAME> --password <PASSWORD>

#Evaluate sql query
For evaluate sql query
> sqoop eval --connect jdbc:mysql://<mysql-URL>:<port>/<DBName> --username <USERNAME> --password <PASSWORD> --query “<SQL QUERY>”


Comments

Post a Comment

Popular posts from this blog

AWS Kinesis - Stream, Firehose, Analytics Overview

AWS Kinesis: AWS Kinesis is managed alternative of Apache Kafka. It can be used for big data real-time stream processing. It can be used for applications logs, metrics, forecast data, IoT. It can be used for streaming processing framework like Spark, NiFi, etc.   Kinesis Capabilities: Kinesis Streams : Streaming data ingest at scale with low latency. It is a data stream. Kinesis Analytics : To perform analytics on real-time streaming data using SQL. You can filter or aggregate data in real time. Kinesis Firehose : To load streams of data into S3, Redshift, Splunk or Elastic Search. It is a delivery stream. Kinesis Data Streams : Streams are divided into shards. To scale up application we can update shard configuration by increasing number of shards. By default shard's data can be retained for 1 Day but you can extend it for 7 days. Multiple application can use same stream. Real-time processing of data with a scale of throughput. Record size shoul...
3 comments
Read more

CCA 175 Preparation with this 6 practice questions- Try to solve it in 1 hour

As I found very few practice questions available on the internet for CCA 175 - Hadoop & spark developer exam. I set 6 questions exam with the solution provided in the comment section.  If you complete it in less than 1 hour then and then think to apply CCA 175 exam else you need more practice. Question's prerequisites : import data from orders table with parquet file format and save data to hdfs path: /user/rj_example/parquet_data/orders import data from customers table and save data to hdfs path: /user/rj_example/data/customers import data from customers table with avro file format and save data to hdfs path: /user/rj_example/avro_data/customers import data from customers table and save data to hdfs path: /user/rj_example/data/categories import data from products table and save data to hdfs path: /user/rj_example/data/products with '\t' as fields seperator create local dir 'rj_example' copy data from /user/rj_example/data/products to local dir ...
3 comments
Read more

Secure Azure Function App with Azure Active Directory (AD). [Token based access]

Welcome to BigDataStacks. This blog is regarding how we can secure azure function app with azure active directory. So when we will try to access function app it will ask for login . I also elaborate on how we can access the function URL with the access token .  Let's start.  Configure Function App Create an Azure Function app with anonymous access. Go to function app's 'Authentication / Authorization' section from 'Platform features'. Turn on App service Authentication/Authorization section. Select action 'Login with Azure AD' Click on Azure AD from Auth provider. Select 'Express' and 'create a new AD app' then click on OK. Click on 'Save'. Again open screen where we selected 'Express mode'. Now Select 'Advanced'. Copy 'clientId' which will be used later. NOTE: If clientId is not showing then refresh the page then it will display.  Add one more entry in 'Allowed Token Audi...
1 comment
Read more