Import and process text file within MySQL
load the file into a temporary/staging table using load data infile and then use a stored procedure to process the data - shouldnt take more than 1-2 mins at the most to completely load and process the data.,Here's my question: is there a way to import the text file into a temporary MySQL table and then use internal MySQL functions (or PHP/Python wrapper) to speed up the processing?,Another thing to consider, with Python you can use multiprocessing to and try parallelize as much as possible. PyMOTW has a good article about multiprocessing., Why is there there an infinite number of possible bases but only a finite number of measurement outcomes
example code (may be of use to you)
truncate table staging; start transaction; load data infile 'your_data.dat' into table staging fields terminated by ',' optionally enclosed by '"' lines terminated by '\n' ( org_name ... ) set org_name = nullif(org_name, ''); commit; drop procedure if exists process_staging_data; delimiter # create procedure process_staging_data() begin insert ignore into organisations(org_name) select distinct org_name from staging; update... etc.. --or use a cursor if you have to ?? end # delimiter; call process_staging_data();
The best tool for exporting a MySQL database to a text file is mysqldump.,Now that you’ve learned how to export a backup of a MySQL database, we’ll explore how to reverse the process and import the backup into an existing database.,With that, you’ve learned how to both export/backup an existing database, and how to then import/restore that data into the same or even a different database.,For example, to export the books database as the book_admin user to the ~/backup/database directory, we might use the following command:
$ mysqldump - u my_username - p database_name > output_file_path
Loading Data into a Table ,MySQL Reference Manual, Creating a Table , Selecting All Data
You could create a text file
containing one record per line, with values separated by tabs,
and given in the order in which the columns were listed in the
CREATE TABLE statement. For
missing values (such as unknown sexes or death dates for animals
that are still living), you can use
values. To represent these in your text file, use
\N (backslash, capital-N). For example, the
record for Whistler the bird would look like this (where the
whitespace between values is a single tab character):
Whistler Gwen bird\ N 1997 - 12 - 09\ N
mariadb-import/mysqlimport, Restoring Data from Dump Files ,If the LOW_PRIORITY flag isn't included, the table will be locked temporarily during the import and other users will be prevented from accessing it.,The number of lines for MariaDB to ignore can, of course, be more than one.
The LOAD DATA INFILE statement is the easiest way to import data from a plain text file into MariaDB. Below is what one would enter in the mysql client to load the data in the file called prospects.txt into the table
LOAD DATA INFILE '/tmp/prospects.txt' INTO TABLE prospect_contact FIELDS TERMINATED BY '|';
Other "within-import" queries related to "Import and process text file within MySQL"
- How to allow only numbers in textbox in reactjs?
- Why we use 'import' in React.js but 'require' in express? [duplicate]
- IO icons from “react icons” not importing
- Typescript cannot find module when import svg file
- How to mock a module import with Sinon and ReactJS
- ./src/components/styles.js Attempted import error: '@material-ui/core/styles' does not contain a default export (imported as 'makeStyles')
- How to change a svg image along with text with hover in CSS
- React Material UI Label Overlaps with Text
- Calling setstate after processing forEach within useEffect
- Dynamically import React component from external URL and use parent React environment
- WebStorm not detecting React import unless explicitly typed
- Reactjs doesnot allow dir=“rtl” to show text RTL
- Move a specific part of text to next line if it have to
- HTML5 text area to only accept numbers
- Attempted import error: 'alpha' is not exported from '@material-ui/core/styles'
- How to get heading text on click in reactJs?
- How to hide text in input field as you type each character?
- Migrating from useContext + useReducer to Redux
- Attempted import error: 'makeObservable' is not exported from 'mobx'
- Heading text not getting on button click in reactJs?
- React native why is my textinput text in the middle?
- Material-UI Core TextField globally set InputLabelProps shrink
- Reactjs update context api data after axios response
- Axios post request error with 422 (Unprocessable Entity) from react
- Overlay text on a picture with PIL
- Remove list entries that match any entry in another list
- Best way to handle different configuration/settings based on environment in Django project
- How to implement an undo()-Method for a List-Class in Python
- Custom enum creation in Python [duplicate]
- Sqlite python insert
- Python double quotes in subprocess.Popen aren't working when executing WinSCP scripting
- Speed of "sum" comprehension in Python
- Time stretching signal / porting python code to matlab / octave
- How to use kafka on tornado?
- Python SUDS - Interrogating the WSDL for MinOccurs and MaxOccurs values
- Python: apply list of functions to each element in list
- Boring Factorials in python
- Scraping data with Python and Pandas
- Mixed Integer Programming - Warehouse Location (Python + GLPK)
- Regular expression to exclude dash and underscore as last character
- Replace elements in a list of lists python
- ElementTree's iterparse() XML parsing error
- What is a relatively simple way to determine the probability that a sentence is in English?
- In Boto3, how to create a Paginator for list_objects with additional keyword arguments?