vt7.0.1 CSV Import no longer works

Worked in version 7.0.0

Clean install, tried several installations, also tried different csv, tried exporting from 7.0.0 and importing always get the same result 0/0 and a success message

@alex.kaplun Can you please share CSV or please enable display_errors 'On' in your local instance

I am seeing the same problem in a brand new clean install of vtiger 7.0.1.

I guess this is the problem:

Mon Jun 12 10:17:06 2017,592 [6693] DEBUG VT - query being executed : CREATE TABLE vtiger_import_1 (id INT PRIMARY KEY AUTO_INCREMENT, status INT DEFAULT 0, recordid INT,accountname varchar(100),
phone varchar(30),otherphone varchar(30),fax varchar(30),bill_street varchar(250),bill_pobox varchar(30),bill_code varchar(30),bill_city varchar(30),bill_country varchar(30)) ENGINE=MyISAM 
Mon Jun 12 10:17:06 2017,635 [6693] DEBUG VT - Prepared sql query being executed : LOAD DATA LOCAL INFILE "/var/www/tmp/modules/Import/helpers/../../../cache/import/IMPORT_1" INTO TABLE vti
ger_import_1 FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY "\"" LINES TERMINATED BY "\n" IGNORE 1 LINES (accountname,phone,otherphone,fax,bill_street,bill_pobox,bill_code,bill_city,bill_country)
Mon Jun 12 10:17:06 2017,635 [6693] INFO VT - PearDatabase ->TRANS creating new connection
Mon Jun 12 10:17:06 2017,635 [6693] INFO VT - PearDatabase ->ADODB error  Query Failed:LOAD DATA LOCAL INFILE "/var/www/tmp/modules/Import/helpers/../../../cache/import/IMPORT_1" INTO TABLE vtiger_import_1 FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY "\"" LINES TERMINATED BY "\n" IGNORE 1 LINES (accountname,phone,otherphone,fax,bill_street,bill_pobox,bill_code,bill_city,bill_country)::->[1148]The used command is not allowed with this MySQL version
Mon Jun 12 10:17:06 2017,635 [6693] DEBUG VT - Prepared sql query being executed : SELECT count(*) AS count FROM vtiger_import_1
Mon Jun 12 10:17:06 2017,636 [6693] DEBUG VT - Prepared sql query being executed : SHOW TABLES LIKE ?
Mon Jun 12 10:17:06 2017,636 [6693] DEBUG VT - Prepared sql query parameters : [vtiger_import_queue]
Mon Jun 12 10:17:06 2017,659 [6693] DEBUG VT - Prepared sql query being executed : INSERT INTO vtiger_import_queue VALUES(?,?,?,?,?,?,?,?,?,?)
Mon Jun 12 10:17:06 2017,659 [6693] DEBUG VT - Prepared sql query parameters : [2,1,6,{"accountname":"0","phone":"1","otherphone":"2","fax":"3","bill_street":"4","bill_pobox":"5","bill_code":"6","bill_city":"7","bill_country":"8"},[],0,"",0,,0]

And as @alex.kaplun said - I can also confirm that this was definitely working previously with version 7.0.0 (and various pulls from the gitlab repo during the development process). I am trying this on the same server as before too with same version of PHP/MySQL etc...

The file to import is being created:

alanlord@dev:/var/www/tmp/cache/import$ ls -l
total 12
-rw-r--r-- 1 www-data www-data 5640 Jun 12 12:35 IMPORT_1

And it gets deleted very quickly and nothing is imported due to the MySQL error above.

@lord_alan, @alex.kaplun which db_type you pointed in your instance(config.inc.php $dbConfig)

@satish.dvnk

$dbconfig['db_type'] = 'mysqli';

Yes @lord_alan, it is reproducible when I make db_type as 'mysqli'. Let me investigate why mysqli is causing this issue.

I've also exported a manually saved organization export.csv, importing behaves the same way attached csv

I also have $dbconfig['db_type'] = 'mysqli';

@satish.dvnk Any developments on this one yet?

@lord_alan the following will fix this issue db200fe4

@satish.dvnk Thanks. I have tried it and that fixes this particular problem. I have also made a comment on your commit as I am not sure it is the right/best way to address it.

@satish.dvnk However, I am getting some weird behaviour in the User Interface itself. Sometimes the imports works. Sometimes the modal overlay doesn't clear afterwards... I am not sure what is happening yet - I will try to describe it better.

@satish.dvnk There are definitely still some issues with import...

Sometimes it simply doesn't work, the jQuery modal overlay doesn't clear, sometimes it just returns very quickly without importing.... It's hard to pin down but this is definitely not right. It is unreliable.

I have one test system where I was able to - eventually - import some Organisations, after several attempts. But now I am unable to import any Contact or Leads :-(

These are csv files I have used for ages as test data and I know they work so I am at a bit of a loss as to why these problems are present.

I will look into it @lord_alan. If you can provide some more details like

• PHP Version:

• Mysql Version:

• Mysql server type:

• Operating system:

If possible, please have a look at this.

This is on one of my dev servers - pretty old. It's Ubuntu 12.04 LTS:

PHP 5.3.10-1ubuntu3.25 with Suhosin-Patch (cli) (built: Oct 3 2016 17:02:21) mysql Ver 14.14 Distrib 5.5.54, for debian-linux-gnu (x86_64) using readline 6.2 mysqli

But I would say the issues seem to me to be more related to the front-end/Javascript rather than the backend.

Thanks @lord_alan for providing info. We will check this and let you know.

What about the duplicate record creation issue which existed in 6.5.0 imports? Is that taken care of in Vtiger 7?

Can you elaborate that issue @nilay.automatesmb or can you mention the issue id

@satish.dvnk unable to locate the issue.

But here it is, if we have a scheduled import. It creates duplicate records, especially if we have large data to import where import could go one for more than 15 min. As while the first CRON run is not completed processing, the next cron job runs after 15 min. And restarts importing records which have been already imported in the previous run.

mentioned in merge request !239 (merged)