DB deploy with mvn fails on osx

[rohityadavcloud]

The mvn deploydb command fails with:

========> Processing upgrade: com.cloud.upgrade.DatabaseUpgradeChecker
[WARNING]
java.lang.reflect.InvocationTargetException
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:297)
    at java.lang.Thread.run (Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to parse systemVM template metadata file: ./engine/schema/dist/systemvm-templates/metadata.ini
    at com.cloud.upgrade.SystemVmTemplateRegistration.parseMetadataFile (SystemVmTemplateRegistration.java:705)
    at com.cloud.upgrade.DatabaseUpgradeChecker.check (DatabaseUpgradeChecker.java:372)
    at com.cloud.upgrade.DatabaseCreator.main (DatabaseCreator.java:219)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:297)
    at java.lang.Thread.run (Thread.java:829)
Caused by: java.lang.NullPointerException
    at com.cloud.upgrade.SystemVmTemplateRegistration.parseMetadataFile (SystemVmTemplateRegistration.java:695)
    at com.cloud.upgrade.DatabaseUpgradeChecker.check (DatabaseUpgradeChecker.java:372)
    at com.cloud.upgrade.DatabaseCreator.main (DatabaseCreator.java:219)

Further, the content of the metadata.ini is:

 cat ./engine/schema/dist/systemvm-templates/metadata.ini

[default]
version = 4.16.1.0

[0]
templatename = systemvm-0-4.16.1
checksum =
downloadurl = https://download.cloudstack.org/systemvm/4.16/systemvmtemplate-4.16.1-ovm.raw.bz2
filename = systemvmtemplate-4.16.1-ovm.raw.bz2

Starting jetty fails with:

ERROR [c.c.u.SystemVmTemplateRegistration] (main:null) (logid:) Failed to parse systemVM template metadata file: ./engine/schema/dist/systemvm-templates/metadata.ini
java.lang.NullPointerException
	at com.cloud.upgrade.SystemVmTemplateRegistration.parseMetadataFile(SystemVmTemplateRegistration.java:695)
	at com.cloud.upgrade.DatabaseUpgradeChecker.check(DatabaseUpgradeChecker.java:372)
	at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.checkIntegrity(CloudStackExtendedLifeCycle.java:64)
	at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:54)

ISSUE TYPE

• Bug Report

COMPONENT NAME

Building/running mgmt server on osx

[rohityadavcloud]

@Pearl1594 I'll need some advise on how the metadata.ini should be created? Perhaps we're doing something in scripts that assumes the OS is Linux?

[weizhouapache]

@Pearl1594 I'll need some advise on how the metadata.ini should be created? Perhaps we're doing something in scripts that assumes the OS is Linux?

like below ?

bash -x engine/schema/templateConfig.sh 4.16.1.0

4.16.1.0 is the value of project.systemvm.template.version in pom.xml

[rohityadavcloud]

This is output of command shared by Wei:

> bash -x engine/schema/templateConfig.sh 4.16.1.0

+ declare -A templates
engine/schema/templateConfig.sh: line 71: declare: -A: invalid option
declare: usage: declare [-afFirtx] [-p] [name[=value] ...]
+ getTemplateVersion 4.16.1.0
+ projVersion=4.16.1.0
++ cut -d- -f1
+ version=4.16.1.0
++ cut -d. -f1
+ subversion1=4
++ cut -d. -f2
+ subversion2=16
++ cut -d. -f3
+ minorversion=1
++ cut -d. -f4
+ securityversion=0
+ export CS_VERSION=4.16
+ CS_VERSION=4.16
+ export CS_MINOR_VERSION=1
+ CS_MINOR_VERSION=1
+ export VERSION=4.16.1
+ VERSION=4.16.1
+ templates=(["kvm"]="https://download.cloudstack.org/systemvm/${CS_VERSION}/systemvmtemplate-$VERSION-kvm.qcow2.bz2" ["vmware"]="https://download.cloudstack.org/systemvm/${CS_VERSION}/systemvmtemplate-$VERSION-vmware.ova" ["xenserver"]="https://download.cloudstack.org/systemvm/$CS_VERSION/systemvmtemplate-$VERSION-xen.vhd.bz2" ["hyperv"]="https://download.cloudstack.org/systemvm/$CS_VERSION/systemvmtemplate-$VERSION-hyperv.vhd.zip" ["lxc"]="https://download.cloudstack.org/systemvm/$CS_VERSION/systemvmtemplate-$VERSION-kvm.qcow2.bz2" ["ovm3"]="https://download.cloudstack.org/systemvm/$CS_VERSION/systemvmtemplate-$VERSION-ovm.raw.bz2")
+++ dirname engine/schema/templateConfig.sh
++ cd -- engine/schema
++ pwd -P
+ PARENTPATH=/Users/rohit/lab/apache/cloudstack/engine/schema/dist/systemvm-templates/
+ mkdir -p /Users/rohit/lab/apache/cloudstack/engine/schema/dist/systemvm-templates/
+ METADATAFILE=/Users/rohit/lab/apache/cloudstack/engine/schema/dist/systemvm-templates/metadata.ini
+ echo
+ SOURCEFILE=/Users/rohit/lab/apache/cloudstack/engine/schema/dist/systemvm-templates/md5sum.txt
+ createMetadataFile
++ cat /Users/rohit/lab/apache/cloudstack/engine/schema/dist/systemvm-templates/md5sum.txt
+ local 'fileData=88fe0da9ebaa27d710af708131b9bf32  systemvmtemplate-4.16.0-hyperv.vhd.zip
2f3747a597396d4ee6a2605648808d1a  systemvmtemplate-4.16.0-kvm.qcow2.bz2
38f84087c5d96548c50a2b52a2856fef  systemvmtemplate-4.16.0-ovm.raw.bz2
606db78c5d138e96d53b4699e77789b2  systemvmtemplate-4.16.0-vmware.ova
6015c93583c7388e6b3e0e7bc6e3517c  systemvmtemplate-4.16.0-xen.vhd.bz2
2d21855fe179a7ce214ffc46c77fac8d  systemvmtemplate-4.16.1-hyperv.vhd.zip
5108eeb0547912a41cb2bc508985f89f  systemvmtemplate-4.16.1-kvm.qcow2.bz2
d889b6e490018360f93e54240d74bf93  systemvmtemplate-4.16.1-ovm.raw.bz2
f55d3b858b708f3103660d83021cb702  systemvmtemplate-4.16.1-vmware.ova
3ccf327c1b0a9c96c506547bf1a6869a  systemvmtemplate-4.16.1-xen.vhd.bz2'
+ echo -e '[default]\nversion = 4.16.1.0\n'
+ for i in '"${!templates[@]}"'
+ section=0
++ getGenericName 0
+++ echo 0
+++ tr '[:upper:]' '[:lower:]'
++ hypervisor=0
++ [[ 0 == \o\v\m\3 ]]
++ [[ 0 == \l\x\c ]]
++ [[ 0 == \x\e\n\s\e\r\v\e\r ]]
++ echo 0
+ hvName=0
+ templatename=systemvm-0-4.16.1
++ getChecksum '88fe0da9ebaa27d710af708131b9bf32  systemvmtemplate-4.16.0-hyperv.vhd.zip
2f3747a597396d4ee6a2605648808d1a  systemvmtemplate-4.16.0-kvm.qcow2.bz2
38f84087c5d96548c50a2b52a2856fef  systemvmtemplate-4.16.0-ovm.raw.bz2
606db78c5d138e96d53b4699e77789b2  systemvmtemplate-4.16.0-vmware.ova
6015c93583c7388e6b3e0e7bc6e3517c  systemvmtemplate-4.16.0-xen.vhd.bz2
2d21855fe179a7ce214ffc46c77fac8d  systemvmtemplate-4.16.1-hyperv.vhd.zip
5108eeb0547912a41cb2bc508985f89f  systemvmtemplate-4.16.1-kvm.qcow2.bz2
d889b6e490018360f93e54240d74bf93  systemvmtemplate-4.16.1-ovm.raw.bz2
f55d3b858b708f3103660d83021cb702  systemvmtemplate-4.16.1-vmware.ova
3ccf327c1b0a9c96c506547bf1a6869a  systemvmtemplate-4.16.1-xen.vhd.bz2' 4.16.1-0
++ local 'fileData=88fe0da9ebaa27d710af708131b9bf32  systemvmtemplate-4.16.0-hyperv.vhd.zip
2f3747a597396d4ee6a2605648808d1a  systemvmtemplate-4.16.0-kvm.qcow2.bz2
38f84087c5d96548c50a2b52a2856fef  systemvmtemplate-4.16.0-ovm.raw.bz2
606db78c5d138e96d53b4699e77789b2  systemvmtemplate-4.16.0-vmware.ova
6015c93583c7388e6b3e0e7bc6e3517c  systemvmtemplate-4.16.0-xen.vhd.bz2
2d21855fe179a7ce214ffc46c77fac8d  systemvmtemplate-4.16.1-hyperv.vhd.zip
5108eeb0547912a41cb2bc508985f89f  systemvmtemplate-4.16.1-kvm.qcow2.bz2
d889b6e490018360f93e54240d74bf93  systemvmtemplate-4.16.1-ovm.raw.bz2
f55d3b858b708f3103660d83021cb702  systemvmtemplate-4.16.1-vmware.ova
3ccf327c1b0a9c96c506547bf1a6869a  systemvmtemplate-4.16.1-xen.vhd.bz2'
++ local hvName=4.16.1-0
++ IFS=
++ read -r line
++ [[ 88fe0da9ebaa27d710af708131b9bf32  systemvmtemplate-4.16.0-hyperv.vhd.zip == *\4\.\1\6\.\1\-\0* ]]
++ IFS=
++ read -r line
++ [[ 2f3747a597396d4ee6a2605648808d1a  systemvmtemplate-4.16.0-kvm.qcow2.bz2 == *\4\.\1\6\.\1\-\0* ]]
++ IFS=
++ read -r line
++ [[ 38f84087c5d96548c50a2b52a2856fef  systemvmtemplate-4.16.0-ovm.raw.bz2 == *\4\.\1\6\.\1\-\0* ]]
++ IFS=
++ read -r line
++ [[ 606db78c5d138e96d53b4699e77789b2  systemvmtemplate-4.16.0-vmware.ova == *\4\.\1\6\.\1\-\0* ]]
++ IFS=
++ read -r line
++ [[ 6015c93583c7388e6b3e0e7bc6e3517c  systemvmtemplate-4.16.0-xen.vhd.bz2 == *\4\.\1\6\.\1\-\0* ]]
++ IFS=
++ read -r line
++ [[ 2d21855fe179a7ce214ffc46c77fac8d  systemvmtemplate-4.16.1-hyperv.vhd.zip == *\4\.\1\6\.\1\-\0* ]]
++ IFS=
++ read -r line
++ [[ 5108eeb0547912a41cb2bc508985f89f  systemvmtemplate-4.16.1-kvm.qcow2.bz2 == *\4\.\1\6\.\1\-\0* ]]
++ IFS=
++ read -r line
++ [[ d889b6e490018360f93e54240d74bf93  systemvmtemplate-4.16.1-ovm.raw.bz2 == *\4\.\1\6\.\1\-\0* ]]
++ IFS=
++ read -r line
++ [[ f55d3b858b708f3103660d83021cb702  systemvmtemplate-4.16.1-vmware.ova == *\4\.\1\6\.\1\-\0* ]]
++ IFS=
++ read -r line
++ [[ 3ccf327c1b0a9c96c506547bf1a6869a  systemvmtemplate-4.16.1-xen.vhd.bz2 == *\4\.\1\6\.\1\-\0* ]]
++ IFS=
++ read -r line
+ checksum=
+ downloadurl=https://download.cloudstack.org/systemvm/4.16/systemvmtemplate-4.16.1-ovm.raw.bz2
++ echo systemvmtemplate-4.16.1-ovm.raw.bz2
+ filename=systemvmtemplate-4.16.1-ovm.raw.bz2
+ echo -e '[0]\ntemplatename = systemvm-0-4.16.1\nchecksum = \ndownloadurl = https://download.cloudstack.org/systemvm/4.16/systemvmtemplate-4.16.1-ovm.raw.bz2\nfilename = systemvmtemplate-4.16.1-ovm.raw.bz2\n'

It appears the bash script is not platform agnostic.

[Pearl1594]

This seems to be an issue with the bash version. The following : "${!templates[@]}" way of referencing keys of a map in bash is supported from version 4 onward

[rohityadavcloud]

@Pearl1594 output on osx:

> bash -version
GNU bash, version 3.2.57(1)-release (arm64-apple-darwin21)
Copyright (C) 2007 Free Software Foundation, Inc.

[rohityadavcloud]

@Pearl1594 is there a way to make this simpler to work on bash 3.x (for osx), like https://stackoverflow.com/questions/1494178/how-to-define-hash-tables-in-bash ?

[rohityadavcloud]

Closing this as we've a PR and PR has been tested manually on osx. #6069