Skip to content
Snippets Groups Projects

vt7.0.1 CSV Import no longer works

    • Closed Issue created by Chris Thompson @Chris.thompson

    Please see ticket #588 (closed) which is no longer appearing in the list of issues and cannot be related here but I can still find from #618 (closed). The import process still appears to not be working in 7.01.

    Designs

      Child items 0

      2. No child items are currently assigned. Use child items to break down this issue into smaller parts.

      Activity

      • C
        Chris Thompson @Chris.thompson ·
        Author

        I've added comments to #618 (closed) if you would like more details no this issue.

      • C
        Chris Thompson @Chris.thompson ·
        Author

        Is anyone monitoring these issues?

      • M
        Manuel @manuelgit ·
        Contributor

        @Chris.thompson seems not. It´s again dead...

      • Alan Lord
        Alan Lord @lord_alan ·
        Contributor

        @satish.dvnk, @prasad, Import is still really messed up. I just installed a clean 7.0.1 (pulled from the master branch) to some dev work. I needed to import some leads.

        The first time I tried it didn't work at all. Just came back after a few seconds to the import screen.

        The second time, nothing happened twice when I clicked import, but when I clicked it for a third time (without closing the Import overlay) it looks like it is trying to import twice :-) I currently see:

        Screenshot_from_2017-09-28_09-51-26

        There are only 355 records in the csv :-)

        And the screen has stayed like the above for about 10 minutes now... It is not refreshing.

      • Alan Lord
        Alan Lord @lord_alan ·
        Contributor

        @Chris.thompson The problem specifically in #588 (closed) was addressed by the patch. #618 (closed) was almost certainly nothing to do with import at all.

        However Import is still broken, in some kind of intermittent way, as I have mentioned above and in #588 (closed) before.

      • Alan Lord
        Alan Lord @lord_alan ·
        Contributor

        Just to update this - I now have 710 Leads imported. From a file which held 355. Because I clicked on an unresponsive button twice (or maybe more). Import is most definitely unreliable at best, and badly broken at worst.

      • M
        Manuel @manuelgit ·
        Contributor

        @satish.dvnk @prasad something is really really wrong with import records.

        To reproduce, just create one record, lets say Contact and export and try to import. No records are imported

        Even says: Total records scanned : 0

        Records successfully imported : 0 / 0 Records created : 0 Records overwritten : 0 Records skipped : 0 Records merged : 0

      • Alan Lord
        Alan Lord @lord_alan ·
        Contributor

        +1. I see this a lot. But try it a few times though and it sometimes does work - eventually.

      • M
        Manuel @manuelgit ·
        Contributor

        @lord_alan well, tried 15 times and didn´t work.

        And why the order of the menu modules on my clean testing installations are always different from the vtiger demos? losing confidence here!!!

        also import mess with text accents:

        Check:

        Fernando Lourenço

        https://demo.vtiger.com/wip/index.php?module=Contacts&view=Detail&record=10&app=SUPPORT

      • P
        Prasad @prasad ·
        Maintainer

        @manuelgit - take a break and share the CSV you tried to import. Also, did you turn on db debug to trace the issue?

      • M
        Manuel @manuelgit ·
        Contributor

        @prasad thanks but already had a coffee (one more)!

        Check, just exported from demo vtiger wip:

        You can see that doesn´t import accents.

        Contacts.csv

      • Prasad Milestone changed to 7.1.0

        Milestone changed to 7.1.0

      • P
        Prasad @prasad ·
        Maintainer

        @manuelgit - checked locally and set the milestone. It's 1:00 AM here... so will get back tomorrow on this.

      • M
        Manuel @manuelgit ·
        Contributor

        @prasad also the log:

        Tue Jan 30 19:46:56 2018,884 [8297] DEBUG VT - Prepared sql query being executed : select * from vtiger_ws_fieldtype
Tue Jan 30 19:46:56 2018,887 [8297] DEBUG VT - query being executed : CREATE TABLE vtiger_import_1 (id INT PRIMARY KEY AUTO_INCREMENT, status INT DEFAULT 0, recordid INT,salutationtype varchar(200),firstname varchar(40),lastname varchar(80),phone varchar(50),account_id varchar(250),mobile varchar(50),leadsource varchar(200),homephone varchar(50),title varchar(50),otherphone varchar(50),department varchar(30),fax varchar(50),email varchar(100),birthday varchar(250),assistant varchar(30),contact_id varchar(250),assistantphone varchar(50),secondaryemail varchar(100),emailoptout varchar(3),donotcall varchar(3),reference varchar(3),assigned_user_id varchar(250),createdtime varchar(250),notify_owner varchar(3),isconvertedfromlead varchar(3),portal varchar(3),support_start_date varchar(250),support_end_date varchar(250),mailingstreet varchar(250),otherstreet varchar(250),mailingpobox varchar(30),otherpobox varchar(30),mailingcity varchar(40),othercity varchar(40),mailingstate varchar(30),otherstate varchar(50),mailingzip varchar(30),otherzip varchar(30),mailingcountry varchar(40),othercountry varchar(30),imagename varchar(150),description mediumtext) ENGINE=MyISAM 
Tue Jan 30 19:46:56 2018,890 [8297] DEBUG VT - Prepared sql query being executed : LOAD DATA LOCAL INFILE "/home/vt71/modules/Import/helpers/../../../cache/import/IMPORT_1" INTO TABLE vtiger_import_1 FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY "\"" LINES TERMINATED BY "\n" IGNORE 1 LINES (salutationtype,firstname,@ignore,lastname,phone,account_id,mobile,leadsource,homephone,title,otherphone,department,fax,email,birthday,assistant,contact_id,assistantphone,secondaryemail,emailoptout,donotcall,reference,assigned_user_id,createdtime,notify_owner,@ignore,@ignore,isconvertedfromlead,@ignore,portal,support_start_date,support_end_date,mailingstreet,otherstreet,mailingpobox,otherpobox,mailingcity,othercity,mailingstate,otherstate,mailingzip,otherzip,mailingcountry,othercountry,imagename,description)
Tue Jan 30 19:46:56 2018,890 [8297] INFO VT - PearDatabase ->TRANS creating new connection
Tue Jan 30 19:46:56 2018,891 [8297] INFO VT - PearDatabase ->ADODB error  Query Failed:LOAD DATA LOCAL INFILE "/home/vt71/modules/Import/helpers/../../../cache/import/IMPORT_1" INTO TABLE vtiger_import_1 FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY "\"" LINES TERMINATED BY "\n" IGNORE 1 LINES (salutationtype,firstname,@ignore,lastname,phone,account_id,mobile,leadsource,homephone,title,otherphone,department,fax,email,birthday,assistant,contact_id,assistantphone,secondaryemail,emailoptout,donotcall,reference,assigned_user_id,createdtime,notify_owner,@ignore,@ignore,isconvertedfromlead,@ignore,portal,support_start_date,support_end_date,mailingstreet,otherstreet,mailingpobox,otherpobox,mailingcity,othercity,mailingstate,otherstate,mailingzip,otherzip,mailingcountry,othercountry,imagename,description)::->[1148]The used command is not allowed with this MariaDB version
Tue Jan 30 19:46:56 2018,891 [8297] DEBUG VT - Prepared sql query being executed : SELECT count(*) AS count FROM vtiger_import_1
Tue Jan 30 19:46:56 2018,892 [8297] DEBUG VT - Prepared sql query being executed : SHOW TABLES LIKE ?
Tue Jan 30 19:46:56 2018,893 [8297] DEBUG VT - Prepared sql query parameters : [vtiger_import_queue]
Tue Jan 30 19:46:56 2018,894 [8297] DEBUG VT - Prepared sql query being executed : INSERT INTO vtiger_import_queue VALUES(?,?,?,?,?,?,?,?,?,?)
Tue Jan 30 19:46:56 2018,894 [8297] DEBUG VT - Prepared sql query parameters : [9,1,4,{"salutationtype":"0","firstname":"1","lastname":"3","phone":"4","account_id":"5","mobile":"6","leadsource":"7","homephone":"8","title":"9","otherphone":"10","department":"11","fax":"12","email":"13","birthday":"14","assistant":"15","contact_id":"16","assistantphone":"17","secondaryemail":"18","emailoptout":"19","donotcall":"20","reference":"21","assigned_user_id":"22","createdtime":"23","notify_owner":"24","isconvertedfromlead":"27","portal":"29","support_start_date":"30","support_end_date":"31","mailingstreet":"32","otherstreet":"33","mailingpobox":"34","otherpobox":"35","mailingcity":"36","othercity":"37","mailingstate":"38","otherstate":"39","mailingzip":"40","otherzip":"41","mailingcountry":"42","othercountry":"43","imagename":"44","description":"45"},[],0,"",0,,0]
Tue Jan 30 19:46:56 2018,895 [8297] DEBUG VT - Prepared sql query being executed : SHOW TABLES LIKE ?
Tue Jan 30 19:46:56 2018,896 [8297] DEBUG VT - Prepared sql query parameters : [vtiger_import_queue]
Tue Jan 30 19:46:56 2018,896 [8297] DEBUG VT - Prepared sql query being executed : SELECT * FROM vtiger_import_queue WHERE tabid=? AND userid=?
Tue Jan 30 19:46:56 2018,896 [8297] DEBUG VT - Prepared sql query parameters : [4,1]
Tue Jan 30 19:46:56 2018,908 [8297] DEBUG VT - Prepared sql query being executed : SHOW TABLES LIKE ?
Tue Jan 30 19:46:56 2018,908 [8297] DEBUG VT - Prepared sql query parameters : [vtiger_import_locks]
Tue Jan 30 19:46:56 2018,909 [8297] DEBUG VT - Prepared sql query being executed : SELECT * FROM vtiger_import_locks WHERE tabid=?
Tue Jan 30 19:46:56 2018,909 [8297] DEBUG VT - Prepared sql query parameters : [4]
Tue Jan 30 19:46:56 2018,909 [8297] DEBUG VT - Prepared sql query being executed : SHOW TABLES LIKE ?
Tue Jan 30 19:46:56 2018,909 [8297] DEBUG VT - Prepared sql query parameters : [vtiger_import_locks]
Tue Jan 30 19:46:56 2018,911 [8297] DEBUG VT - Prepared sql query being executed : INSERT INTO vtiger_import_locks VALUES(?,?,?,?,?)
Tue Jan 30 19:46:56 2018,911 [8297] DEBUG VT - Prepared sql query parameters : [9,1,4,9,2018-01-30 19:46:56]
Tue Jan 30 19:46:56 2018,916 [8297] DEBUG VT - Entering getColumnFields(Contacts) method ...
Tue Jan 30 19:46:56 2018,916 [8297] DEBUG VT - in getColumnFields Contacts
Tue Jan 30 19:46:56 2018,917 [8297] DEBUG VT - Exiting getColumnFields method ...
Tue Jan 30 19:46:56 2018,917 [8297] DEBUG VT - Entering getColumnFields(Contacts) method ...
Tue Jan 30 19:46:56 2018,917 [8297] DEBUG VT - in getColumnFields Contacts
Tue Jan 30 19:46:56 2018,917 [8297] DEBUG VT - Exiting getColumnFields method ...
Tue Jan 30 19:46:56 2018,917 [8297] DEBUG VT - Entering getColumnFields(Contacts) method ...
Tue Jan 30 19:46:56 2018,917 [8297] DEBUG VT - in getColumnFields Contacts
Tue Jan 30 19:46:56 2018,917 [8297] DEBUG VT - Exiting getColumnFields method ...
Tue Jan 30 19:46:56 2018,917 [8297] DEBUG VT - Entering getColumnFields(Contacts) method ...
Tue Jan 30 19:46:56 2018,917 [8297] DEBUG VT - in getColumnFields Contacts
Tue Jan 30 19:46:56 2018,917 [8297] DEBUG VT - Exiting getColumnFields method ...
Tue Jan 30 19:46:56 2018,917 [8297] DEBUG VT - Prepared sql query being executed : select * from vtiger_ws_entity where name=?
Tue Jan 30 19:46:56 2018,917 [8297] DEBUG VT - Prepared sql query parameters : [Contacts]
Tue Jan 30 19:46:56 2018,918 [8297] DEBUG VT - Entering getColumnFields(Contacts) method ...
Tue Jan 30 19:46:56 2018,919 [8297] DEBUG VT - in getColumnFields Contacts
Tue Jan 30 19:46:56 2018,919 [8297] DEBUG VT - Exiting getColumnFields method ...
Tue Jan 30 19:46:56 2018,919 [8297] DEBUG VT - Entering getColumnFields(Contacts) method ...
Tue Jan 30 19:46:56 2018,919 [8297] DEBUG VT - in getColumnFields Contacts
Tue Jan 30 19:46:56 2018,919 [8297] DEBUG VT - Exiting getColumnFields method ...
Tue Jan 30 19:46:56 2018,919 [8297] DEBUG VT - Prepared sql query being executed : SELECT DISTINCT block, vtiger_field.tabid, name, blocklabel FROM vtiger_field INNER JOIN vtiger_blocks ON vtiger_blocks.blockid = vtiger_field.block INNER JOIN vtiger_tab ON vtiger_tab.tabid = vtiger_field.tabid WHERE vtiger_tab.name IN (?) AND vtiger_field.presence IN (0,2) ORDER BY block
Tue Jan 30 19:46:56 2018,919 [8297] DEBUG VT - Prepared sql query parameters : [Contacts]
Tue Jan 30 19:46:56 2018,920 [8297] DEBUG VT - Prepared sql query being executed : select *, '0' as readonly from vtiger_field where tabid =? and block in (?,?,?,?,?) and displaytype in (1,2,3,4,5,6)
Tue Jan 30 19:46:56 2018,920 [8297] DEBUG VT - Prepared sql query parameters : [4,4,6,7,8,73]
Tue Jan 30 19:46:56 2018,923 [8297] DEBUG VT - Prepared sql query being executed : SELECT * FROM vtiger_import_1 WHERE status = 0 LIMIT 250
Tue Jan 30 19:46:56 2018,923 [8297] DEBUG VT - Prepared sql query being executed : UPDATE vtiger_import_queue SET status=? WHERE importid=?
Tue Jan 30 19:46:56 2018,923 [8297] DEBUG VT - Prepared sql query parameters : [3,9]
Tue Jan 30 19:46:56 2018,925 [8297] DEBUG VT - Prepared sql query being executed : SHOW TABLES LIKE ?
Tue Jan 30 19:46:56 2018,925 [8297] DEBUG VT - Prepared sql query parameters : [vtiger_import_queue]
Tue Jan 30 19:46:56 2018,926 [8297] DEBUG VT - Prepared sql query being executed : SELECT * FROM vtiger_import_queue WHERE tabid=? AND userid=?
Tue Jan 30 19:46:56 2018,926 [8297] DEBUG VT - Prepared sql query parameters : [4,1]
Tue Jan 30 19:46:56 2018,926 [8297] DEBUG VT - Entering getColumnFields(Contacts) method ...
Tue Jan 30 19:46:56 2018,926 [8297] DEBUG VT - in getColumnFields Contacts
Tue Jan 30 19:46:56 2018,926 [8297] DEBUG VT - Exiting getColumnFields method ...
Tue Jan 30 19:46:56 2018,926 [8297] DEBUG VT - Entering getColumnFields(Contacts) method ...
Tue Jan 30 19:46:56 2018,927 [8297] DEBUG VT - in getColumnFields Contacts
Tue Jan 30 19:46:56 2018,927 [8297] DEBUG VT - Exiting getColumnFields method ...
Tue Jan 30 19:46:56 2018,927 [8297] DEBUG VT - Entering getColumnFields(Contacts) method ...
Tue Jan 30 19:46:56 2018,927 [8297] DEBUG VT - in getColumnFields Contacts
Tue Jan 30 19:46:56 2018,927 [8297] DEBUG VT - Exiting getColumnFields method ...
Tue Jan 30 19:46:56 2018,927 [8297] DEBUG VT - Entering getColumnFields(Contacts) method ...
Tue Jan 30 19:46:56 2018,927 [8297] DEBUG VT - in getColumnFields Contacts
Tue Jan 30 19:46:56 2018,927 [8297] DEBUG VT - Exiting getColumnFields method ...
Tue Jan 30 19:46:56 2018,927 [8297] DEBUG VT - query being executed : SELECT status FROM vtiger_import_1
Tue Jan 30 19:46:56 2018,927 [8297] DEBUG VT - Prepared sql query being executed : SHOW TABLES LIKE ?
Tue Jan 30 19:46:56 2018,927 [8297] DEBUG VT - Prepared sql query parameters : [vtiger_import_locks]
Tue Jan 30 19:46:56 2018,928 [8297] DEBUG VT - Prepared sql query being executed : DELETE FROM vtiger_import_locks WHERE userid=? AND tabid=?
Tue Jan 30 19:46:56 2018,928 [8297] DEBUG VT - Prepared sql query parameters : [1,4]
Tue Jan 30 19:46:56 2018,929 [8297] DEBUG VT - Prepared sql query being executed : SHOW TABLES LIKE ?
Tue Jan 30 19:46:56 2018,929 [8297] DEBUG VT - Prepared sql query parameters : [vtiger_import_queue]
Tue Jan 30 19:46:56 2018,930 [8297] DEBUG VT - Prepared sql query being executed : DELETE FROM vtiger_import_queue WHERE importid=?
Tue Jan 30 19:46:56 2018,930 [8297] DEBUG VT - Prepared sql query parameters : [9]
Tue Jan 30 19:47:11 2018,565 [8346] INFO VT - PearDatabase ->ADODB disconnect
      • Prasad Added ~11 ~10 labels

        Added ~11 ~10 labels

      • P
        Prasad @prasad ·
        Maintainer

        @manuelgit - I did a quick check on instance and here are observations.

        • Step 1: Started with clean Contacts module.
        • Step 2: Import of Contacts CSV shared - 1 record imported.
        • Step 3: Tried to import the Contacts CSV again
        • Step 4: Import of Contacts CSV was 0 (as 2nd Step - Duplicate check was configured to Skip).
        • Step 5: Tried to import the Contacts CSV again (now Skipped Duplicate check altogether - see). 1 record was again imported.

        Source: current master

      • M
        Manuel @manuelgit ·
        Contributor

        @prasad yes, but check this line of the log:

        1148]The used command is not allowed with this MariaDB version

        Tue Jan 30 19:46:56 2018,891 [8297] INFO VT - PearDatabase ->ADODB error Query Failed:LOAD DATA LOCAL INFILE "/home/vt71/modules/Import/helpers/../../../cache/import/IMPORT_1" INTO TABLE vtiger_import_1 FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY """ LINES TERMINATED BY "\n" IGNORE 1 LINES (salutationtype,firstname,@ignore,lastname,phone,account_id,mobile,leadsource,homephone,title,otherphone,department,fax,email,birthday,assistant,contact_id,assistantphone,secondaryemail,emailoptout,donotcall,reference,assigned_user_id,createdtime,notify_owner,@ignore,@ignore,isconvertedfromlead,@ignore,portal,support_start_date,support_end_date,mailingstreet,otherstreet,mailingpobox,otherpobox,mailingcity,othercity,mailingstate,otherstate,mailingzip,otherzip,mailingcountry,othercountry,imagename,description)::->[1148]The used command is not allowed with this MariaDB version

      • M
        Manuel @manuelgit ·
        Contributor

        @prasad you must test this system not just with mysql, are you also testing with mariadb? @lord_alan and what's your database system?

        Because with mariadb i see lots of:

        vtiger_cv2role' doesn't exist

        vtiger_cv2rs' doesn't exist

        Look for this issue #740 (closed) and Here but seems no one is paying attention: #583 (closed)

      • Alan Lord
        Alan Lord @lord_alan ·
        Contributor

        MySQL 5.5 on one particular dev server (Ubuntu 12.04) but I also see this on customers' servers; mostly Ubuntu 14.04 or 16.04 but I think always MySQL. I only know of just one customer system we have running on MariaDB and that is 6.3.0.

      • P
        Prasad @prasad ·
        Maintainer

        @manuelgit - Accent issue during import got fixed with #778 (closed)

      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      • Loading
      Please register or sign in to reply

      Copyright 2023 Vtiger. All rights reserved.