Generic vSphere Commands for Troubleshooting .

what is vSphere command ?

The vSphere Command-Line Interface (vSphere CLI) command set allows you to run common system administration commands against ESXi systems from any machine with network access to those systems.

You have to run below commands in ESXi shell, vCLI, SSH or within the vSphere Management Assistant (vMA) .

Please explorer below vSphere generic commands ,log-path and tool , For knowledge sharing you may use VMware KB articles and reference link :

• vSphere Management Assistant appliance – Recommended, safest way to execute commands
  • vCLI commands
    • esxcli-* commands
      • Primary set of commands to be used for most ESXi host based operations
      • VMware online reference
        • esxcli device 
          • Lists descriptions of device commands.
        • esxcli esxcli
          • Lists descriptions of esxcli commands.
        • esxcli fcoe
          • FCOE (Fibre Channel over Ethernet) commands
        • esxcli graphics
          • Graphics commands
        • esxcli hardware
          • Hardware namespace. Used primarily for extracting information about the current system setup.
        • esxcli iscsi
          • iSCSI namespace for monitoring and managing hardware and software iSCSI.
        • esxcli network
          • Network namespace for managing virtual networking including virtual switches and VMkernel network interfaces.
        • esxcli sched
          • Manage the shared system-wide swap space.
        • esxcli software
          • Software namespace. Includes commands for managing and installing image profiles and VIBs.
        • esxcli storage
          • Includes core storage commands and other storage management commands.
        • esxcli system
          • System monitoring and management command.
        • esxcli vm
          • Namespace for listing virtual machines and shutting them down forcefully.
        • esxcli vsan
          • Namespace for VSAN management commands. See the vSphere Storage publication for details.
    • vicfg-* commands
      • Primarily used for managing Storage, Network and Host configuration
      • Can be run against ESXi systems or against a vCenter Server system.
      • If the ESXi system is in lockdown mode, run commands against the vCenter Server
      • Replaces most of the esxcfg-* commands. A direct comparison can be found here
      • VMware online reference
        • vicfg-advcfg
          • Performs advanced configuration including enabling and disabling CIM providers. Use this command as instructed by VMware.
        • vicfg-authconfig
          • Manages Active Directory authentication.
        • vicfg-cfgbackupBacks up the configuration data of an ESXi system and
          • Restores previously saved configuration data.
        • vicfg-dns
          • Specifies an ESX/ESXi host’s DNS configuration.
        • vicfg-dumppart
          • Manages diagnostic partitions.
        • vicfg-hostops
          • Allows you to start, stop, and examine ESX/ESXi hosts and to instruct them to enter maintenance mode and exit from maintenance mode.
        • vicfg-ipsec
          • Supports setup of IPsec.
        • vicfg-iscsi
          • Manages iSCSI storage.
        • vicfg-module
          • Enables VMkernel options. Use this command with the options listed, or as instructed by VMware.
        • vicfg-mpath
          • Displays information about storage array paths and allows you to change a path’s state.
        • vicfg-mpath35
          • Configures multipath settings for Fibre Channel or iSCSI LUNs.
        • vicfg-nas
          • Manages NAS file systems.
        • vicfg-nics
          • Manages the ESX/ESXi host’s NICs (uplink adapters).
        • vicfg-ntp
          • Specifies the NTP (Network Time Protocol) server.
        • vicfg-rescan
          • Rescans the storage configuration.
        • vicfg-route
          • Lists or changes the ESX/ESXi host’s route entry (IP gateway).
        • vicfg-scsidevs
          • Finds available LUNs.
        • vicfg-snmp
          • Manages the Simple Network Management Protocol (SNMP) agent.
        • vicfg-syslog
          • Specifies the syslog server and the port to connect to that server for ESXi hosts.
        • vicfg-user
          • Creates, modifies, deletes, and lists local direct access users and groups of users.
        • vicfg-vmknic
          • Adds, deletes, and modifies virtual network adapters (VMkernel NICs).
        • vicfg-volume
          • Supports resignaturing a VMFS snapshot volume and mounting and unmounting the snapshot volume.
        • vicfg-vswitch
          • Adds or removes virtual switches or vNetwork Distributed Switches, or modifies switch settings.
    • vmware-cmd commands
      • Commands implemented in Perl that do not have a vicfg- prefix.
      • Performs virtual machine operations remotely including creating a snapshot, powering the virtual machine on or off, and getting information about the virtual machine.
      • VMware online reference
        • vmware-cmd <path to the .vmx file> <VM operations>
    • vmkfstools command
      • Creates and manipulates virtual disks, file systems, logical volumes, and physical storage devices on ESXi hosts.
      • VMware online reference
• ESX shell / SSH
  • esxcli-* commandlets
    • Primary set of commands to be used for most ESXi host based operations
    • VMware online reference
      • esxcli device 
        • Lists descriptions of device commands.
      • esxcli esxcli
        • Lists descriptions of esxcli commands.
      • esxcli fcoe
        • FCOE (Fibre Channel over Ethernet) commands
      • esxcli graphics
        • Graphics commands
      • esxcli hardware
        • Hardware namespace. Used primarily for extracting information about the current system setup.
      • esxcli iscsi
        • iSCSI namespace for monitoring and managing hardware and software iSCSI.
      • esxcli network
        • Network namespace for managing virtual networking including virtual switches and VMkernel network interfaces.
      • esxcli sched
        • Manage the shared system-wide swap space.
      • esxcli software
        • Software namespace. Includes commands for managing and installing image profiles and VIBs.
      • esxcli storage
        • Includes core storage commands and other storage management commands.
      • esxcli system
        • System monitoring and management command.
      • esxcli vm
        • Namespace for listing virtual machines and shutting them down forcefully.
      • esxcli vsan
        • Namespace for VSAN management commands. See the vSphere Storage publication for details.
  • esxcfg-* commands (deprecated but still works on ESXi 5.5)
    • VMware online reference
  • vmkfstools command
    • Creates and manipulates virtual disks, file systems, logical volumes, and physical storage devices on ESXi hosts.
    • VMware online reference

• Log File Locations
  • vCenter Log Files
    • Windows version
      • C:\Documents and settings\All users\Application Data\VMware\VMware VirtualCenter\Logs
      • C:\ProgramData\Vmware\Vmware VirtualCenter\Log
    • Appliance version
      • /var/log
    • VMware KB for SSO log files
    • VMware KB for all vCenter log files
  • ESXi Server Logs
    • /var/log (Majority of ESXi log location)
    • /etc/vmware/vpxa/vpxa.cfg (vpxa/vCenter agent configuration file)
    • VMware KB for all ESXi log file locations
    • /etc/opt/VMware/fdm (FDM agent files for HA configuration)
  • Virtual Machine Logs
    • /vmfs/volumes/<directory name>/<VM name>/VMware.log (Virtual machine log file)
    • /vmfs/volumes/<directory name>/<VM name>/<*.vmdk files> (Virtual machine descriptor files with references to CID numbers of itself and parent vmdk files if snapshots exists)
    • /vmfs/volumes/<directory name>/<VM name>/<*.vmx files> (Virtual machine configuration settings including pointers to vmdk files..etc>

• Networking commands (used to identify and fix network configuration issues)
  • Basic network troubleshooting commands
    • ping
      • Good old ping.
      • Need I say anymore?
    • vmkping
      • Ping from vmkernal port groups.
      • Useful for IP storage networking troubleshooting
      • Can ping from a specific vmkernal interface using the “vmkping -I <vmkernal interface name such as vmk0, vmk1….> <host IP>”
      • VMware Online reference API for the command
    • esxcli network * commands
    • vicfg-* commands (network related)
      • vicfg-dns
      • vicfg-nics
      • vicfg-ipsec
      • vicfg-route
      • vicfg-vmknic
      • vicfg-vswitch
  • Physical Hardware Troubleshooting
    • lspci -p
  • Traffic capture commands
    • tcpdump-uw
      • Works with all versions of ESXi
      • Refer to VMware KB for additional information
    • pktcap-uw
      • Only works with ESXi 5.5
      • Refer to VMware KB for additional information
  • Telnet equivilent
    • nc command (netcat)
      • Used to verify that you can reach a certain port on a destination host (similar to telnet)
      • Run on the esxi shell or ssh
      • Example:  nc -z <ip address of iSCSI server> 3260 check if the iSCSI port can be reached from esxi to iSCSI server
      • VMware KB article
  • Network performance related commands
    • esxtop (ESXi Shell or SSH) & resxtop (vCli) – ‘n’ for networking

• Storage Commands (used to identify & fix vaious storage issues)
  • Basic storage commands
    • esxcli storage * commands
    • vmkfstools command
    • vicfg-* commands
      • vicfg-dumppart:  Manages diagnostic partitions.
      • vicfg-iscsi:  Manages iSCSI storage.
      • vicfg-mpath:  Displays information about storage array paths and allows you to change a path’s state.
      • vicfg-mpath35:  Configures multipath settings for Fibre Channel or iSCSI LUNs.
      • vicfg-nas:  Manages NAS file systems.
      • vicfg-rescan:  Rescans the storage configuration.
      • vicfg-scsidevs:  Finds available LUNs.
      • vicfg-volume:  Supports resignaturing a VMFS snapshot volume and mounting and un-mounting the snapshot volume.
  • VMFS metadata inconsistencies
    • voma command (VMware vSphere Ondisk Metadata Analyser)
      • Example: voma -m vmfs -f check -d /vmfs/devices/disks/naa.xxxx:y (where y is the partition number)
      • Refer to VMware KB article for additional information
  • disk space utilisation
    • df command
  • Storage performance related commands
    • esxtop (ESXi Shell or SSH) & resxtop (vCli) – ‘n‘ for networking

• vCenter server commands (used to identify & fix vCenter, SSO, Inventory related issues)
  • Note that most of the commands available here are Windows commands that can be used to troubleshoot these issues which I wont mention here. Only few key VMware vSphere specific commands are mentioned below instead.
  • SSO
    • ssocli command (C:\Program Files\VMware\Infrastructure\SSOServer\utils\ssocli)
  • vCenter
    • vpxd.exe command (C:\Program Files\VMware\Infrastructure\VirtualCenter Server\vpxd.exe)

• Virtual Machine related commands (used to identify & fix VM related issues)
  • Generic VM commands
    • vmware-cmd commands (vCLI only)
    • vmkfstools command
  • File locking issues
    • touch command
      • Example:  touch <file name>
      • VMware KB article
    • vmkfstools -D command
      • Example: vmkfstools -D /vmfs/volumes/<directory name>/<VM name>/<VM Name.vmdk> (shows the MAC address of the ESXi server with the file lock. it its locked by the same esxi server as where the command was run, ‘000000000000’ is shown)
    • lsof command (identifies the process locking the file)
      • Example: lsof | grep <name of the locked file>
    • kill command (kills the process)
      • Example: kill <PID>
    • md5sum command (used to calculate file checksums)

Thanks hope you like it.

Rajiv Pandey.