Import and process text file within MySQL

Asked
Active3 hr before
Viewed126 times

4 Answers

withinimport
90%

load the file into a temporary/staging table using load data infile and then use a stored procedure to process the data - shouldnt take more than 1-2 mins at the most to completely load and process the data.,Here's my question: is there a way to import the text file into a temporary MySQL table and then use internal MySQL functions (or PHP/Python wrapper) to speed up the processing?,Another thing to consider, with Python you can use multiprocessing to and try parallelize as much as possible. PyMOTW has a good article about multiprocessing., Why is there there an infinite number of possible bases but only a finite number of measurement outcomes

example code (may be of use to you)

truncate table staging;

start transaction;

load data infile 'your_data.dat'
into table staging
fields terminated by ','
optionally enclosed by '"'
lines terminated by '\n'
   (
      org_name
      ...
   )
set
org_name = nullif(org_name, '');

commit;

drop procedure
if exists process_staging_data;

delimiter #

create procedure process_staging_data()
begin

insert ignore into organisations(org_name) select distinct org_name from staging;

update...

   etc..

   --or use a cursor
if you have to ??

   end #

delimiter;

call process_staging_data();
88%

The best tool for exporting a MySQL database to a text file is mysqldump.,Now that you’ve learned how to export a backup of a MySQL database, we’ll explore how to reverse the process and import the backup into an existing database.,With that, you’ve learned how to both export/backup an existing database, and how to then import/restore that data into the same or even a different database.,For example, to export the books database as the book_admin user to the ~/backup/database directory, we might use the following command:

$ mysqldump - u my_username - p database_name > output_file_path
load more v
72%

Loading Data into a Table ,MySQL Reference Manual, Creating a Table , Selecting All Data

You could create a text file pet.txt containing one record per line, with values separated by tabs, and given in the order in which the columns were listed in the CREATE TABLE statement. For missing values (such as unknown sexes or death dates for animals that are still living), you can use NULL values. To represent these in your text file, use \N (backslash, capital-N). For example, the record for Whistler the bird would look like this (where the whitespace between values is a single tab character):

Whistler Gwen bird\ N 1997 - 12 - 09\ N
load more v
65%

mariadb-import/mysqlimport, Restoring Data from Dump Files ,If the LOW_PRIORITY flag isn't included, the table will be locked temporarily during the import and other users will be prevented from accessing it.,The number of lines for MariaDB to ignore can, of course, be more than one.

The LOAD DATA INFILE statement is the easiest way to import data from a plain text file into MariaDB. Below is what one would enter in the mysql client to load the data in the file called prospects.txt into the table prospect_contact:

LOAD DATA INFILE '/tmp/prospects.txt'
INTO TABLE prospect_contact
FIELDS TERMINATED BY '|';
load more v

Other "within-import" queries related to "Import and process text file within MySQL"