Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

how to push a csv data to mongodb using python

Trying to push csv data in to mongodb using python.i'm a beginner to python & mongodb..i used the following code

import csv
import json
import pandas as pd
import sys, getopt, pprint
from pymongo import MongoClient
#CSV to JSON Conversion
csvfile = open('C://test//final-current.csv', 'r')
jsonfile = open('C://test//6.json', 'a')
reader = csv.DictReader( csvfile )
header= [ "S.No", "Instrument Name", "Buy Price", "Buy Quantity", "Sell Price", "Sell Quantity", "Last Traded Price", "Total Traded Quantity", "Average Traded Price", "Open Price", "High Price", "Low Price", "Close Price", "V" ,"Time"]
#fieldnames=header
output=[]
for each in reader:
    row={}
    for field in header:
        row[field]=each[field]
    output.append(row)

json.dump(output, jsonfile, indent=None, sort_keys=False , encoding="UTF-8")
mongo_client=MongoClient() 
db=mongo_client.october_mug_talk
db.segment.drop()
data=pd.read_csv('C://test//6.json', error_bad_lines=0)
df = pd.DataFrame(data)
records = csv.DictReader(df)
db.segment.insert(records)

but the output is given in this format

/* 0 */
{
  "_id" : ObjectId("54891c4ffb2a0303b0d43134"),
  "[{\"AverageTradedPrice\":\"0\"" : "BuyPrice:\"349.75\""
}

/* 1 */
{
  "_id" : ObjectId("54891c4ffb2a0303b0d43135"),
  "[{\"AverageTradedPrice\":\"0\"" : "BuyQuantity:\"3000\""
}

/* 2 */
{
  "_id" : ObjectId("54891c4ffb2a0303b0d43136"),
  "[{\"AverageTradedPrice\":\"0\"" : "ClosePrice:\"350\""
}

/* 3 */
{
  "_id" : ObjectId("54891c4ffb2a0303b0d43137"),
  "[{\"AverageTradedPrice\":\"0\"" : "HighPrice:\"0\""
}

Actually i want the output to like for single id all the other fields should be showed as subtypes eg:

 _id" : ObjectId("54891c4ffb2a0303b0d43137")
    AveragetradedPrice :0
    HighPrice:0
    ClosePrice:350
    buyprice:350.75

Please help me Out.Thanks in advance

like image 939
Viswanathan Avatar asked Dec 11 '14 05:12

Viswanathan


People also ask

How do I import a CSV file into MongoDB using Python?

We will CSV file name user using DictReader() method. The DictReader() function returns a csv reader object. We will iterate(loop) over the CSV reader object and create JSON data to insert multiple records into the MongoDB database. Finally, we will insert JSON rows data into MongoDB using insert_many() method.

Can you upload CSV to MongoDB?

You can use the mongoimport command to import CSV files into a collection using the headerline option. Headerline option notifies the mongoimport command of the first line; to be not imported as a document since it contains field names instead of data.

How do I link my CSV file to MongoDB?

If you have CSV files (or TSV files - they're conceptually the same) to import, use the --type=csv or --type=tsv option to tell mongoimport what format to expect. Also important is to know whether your CSV file has a header row - where the first line doesn't contain data - instead it contains the name for each column.


1 Answers

Thank you for the suggestion.This one is the corrected code:

import csv
import json
import pandas as pd
import sys, getopt, pprint
from pymongo import MongoClient
#CSV to JSON Conversion
csvfile = open('C://test//final-current.csv', 'r')
reader = csv.DictReader( csvfile )
mongo_client=MongoClient() 
db=mongo_client.october_mug_talk
db.segment.drop()
header= [ "S No", "Instrument Name", "Buy Price", "Buy Quantity", "Sell Price", "Sell Quantity", "Last Traded Price", "Total Traded Quantity", "Average Traded Price", "Open Price", "High Price", "Low Price", "Close Price", "V" ,"Time"]

for each in reader:
    row={}
    for field in header:
        row[field]=each[field]

    db.segment.insert(row)
like image 84
Viswanathan Avatar answered Oct 22 '22 04:10

Viswanathan