A brief introduction
TL;DR: In a home lab project, a Cisco UCS server was acquired to replace an older Dell server. However, the Cisco server’s fans were excessively loud and power-intensive. By gaining root access to the server’s Cisco Integrated Management Controller (IMC), which is similar to HP’s iLO or Dell’s iDRAC, it was possible to explore and modify the server’s settings. This included discovering and using a tool to control fan speeds, leading to a significant reduction in noise and power consumption. Challenges include ensuring these changes persist after a reboot, which might be achievable through a script on the server’s non-volatile filesystem.
In my home lab, a hub for hosting home automation, media streaming, and various other services, I embarked on a weekend project to upgrade my existing Dell server. Despite its age, the Dell still had potential, but the lure of advanced technology led me to a remarkable find on eBay: a Cisco UCS server, attractively priced and well-spec’d. However, the transition wasn’t smooth. My attempt to match the efficiency and quietness of the Dell with the Cisco UCS was challenging. The Cisco’s fans operated at a relentless 9K RPM, about 50% speed, transforming it into a loud, power-intensive beast. Attempts to reduce the fan speed, including removing a PCIe network card and GPU – flagged as non-Cisco by the IMC – were only partially successful.
This endeavour highlights the challenges of managing a home lab in the UK, where electricity costs are steep at £0.30 per KWh. My servers, like the robust C240, are powerful yet noisy, their fans designed for cooler, data center conditions and high load. Although there are ways to lessen their noise, these are limited in effectiveness. The prospect of a loud, energy-guzzling server, particularly under low load, is unattractive. Additionally, owning such advanced equipment ignites a desire for complete administrative control, or root access, to fully harness its capabilities.
After regretting my new purchase and considering giving up on everything IT, I had a thought. If the Cisco IMC controls the server hardware, can it be manipulated in some way to save me from the wrath of the wife and the awful power hungry fans?
So what is a Cisco IMC?
Cisco’s Integrated Management Controller (IMC) is a dedicated system tailored for monitoring and managing Cisco servers. This comprehensive tool offers remote control, configuration, and health monitoring of server hardware, effectively acting as the server’s “brain,” enabling administrators to manage the servers remotely. Its very similar to a HP iLO or a Dell iDRAC and shares a lot of the underlying technology of those and many more products.
Integrated into all Cisco UCS C-Series and S-Series servers, the IMC offers capabilities comparable to, and sometimes surpassing, direct physical access. In the case of the Cisco IMC, it also controls the server fans and in fact almost every bit of hardware on the server.
How I got root?
I can’t share details about a security exploit yet due to responsible disclosure, which is important to fix vulnerabilities before they’re known to hackers. I’m working with the relevant people to enhance security, and a CVE (security alert) is expected to be published by mid-February 2024, with more information to follow in a separate blog post.
UPDATE 01/02/2024
The release of further details of this vulnerability will be delayed until March as per request of Cisco PSIRT. These things always take longer than you expect!
Now I am root!
[root@C240:~]$id
uid=0(root) gid=0(root) groups=0(root)Once you are root you can finally have a good poke around the system and it turns out that the IMC actually runs a fairly standard embedded Linux build on a 32bit ARM V7 little endian 2 core CPU. While not hugely different to a Raspberry Pi 2 in spec, it only has 256Mb of RAM!
processor : 0
model name : ARMv7 Processor rev 1 (v7l)
BogoMIPS : 997.78
Features : half thumb fastmult vfp edsp vfpv3 vfpv3d16 tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x4
CPU part : 0xc09
CPU revision : 1
processor : 1
model name : ARMv7 Processor rev 1 (v7l)
BogoMIPS : 997.78
Features : half thumb fastmult vfp edsp vfpv3 vfpv3d16 tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x4
CPU part : 0xc09
CPU revision : 1
Hardware : Pilot4 BMC SoC
Revision : 0000
Serial : 0000000000000000[root:~]$cat /proc/meminfo
MemTotal: 233524 kB
MemFree: 12820 kB
MemAvailable: 46612 kB
~The Linux Kernel version is 5.4.132.4 which was released some time after November 2019. There are some heavily customised binaries too, these are quite interesting:
[root@C240:~]$openssl version
CiscoSSL 1.1.1q.7.2.440
FIPS mode: OFF
[root@C240:~]$sshd -V
CiscoSSH 1.10.32, OpenSSH_8.8p1, CiscoSSL 1.1.1q.7.2.440Some interesting files in /proc/cisco
Looks like some are for debugging but mostly they appear to be used to monitor the server hardware.
[root@C240:cisco]$cd /proc/cisco
[root@C240:cisco]$ls -1
bmc-unique-id
bmc-unique-id-text
bt_stat
challenge-counter-binary
chassis_open
chassis_open_boot_sts
enable_snoop_com1
enable_snoop_com2
enable_sol_com1
enable_sol_com2
gpio
gpio_i2c
main_public_rollover_key
mmio_address
mmio_memory
mmio_size
ncsi
panic-on-hung-task-info
pmbus
port_80_capture
power_button_press_counter
scratchNsniff
srv_P_T_state
srv_power_button
stop_reboot_on_oom
timed_port_80_capture
tulip
vdd_power_good
watchdog_resetAnd some in /proc/cisco/gpio
Its amazing how much there is here, again mostly they appear to be for server monitoring functionality.
[root@C240:cisco]$cd /proc/cisco
[root@C240:gpio]$ls -1
12v_stby_cnic1_en_n
12v_stby_cnic2_en_n
12v_stby_flom_en_n
88e6176_int_n
88e6176_int_n-dbg-ctrl
adr_complete_pch
adr_complete_pch-dbg-ctrl
adr_trig_en_ibmc
adr_trigger_ibmc
adr_trigger_pch
adr_trigger_pch-dbg-ctrl
benecia_reset_ctrl
bios_bmc_gpio2
bios_bmc_gpio2-dbg-ctrl
bmc_boot_alt_image
bmc_boot_alt_image-dbg-ctrl
bmc_pch_password_override
bmc_pinhole_btn
bmc_pinhole_btn-dbg-ctrl
bmc_reset_pwd
bmc_reset_pwd-dbg-ctrl
bmc_rfd
bmc_rfd-dbg-ctrl
card0_pcie_detect
card0_pcie_detect-dbg-ctrl
card1_pcie_detect
card1_pcie_detect-dbg-ctrl
cmos_rtcrst
cnic1_prsnt_n
cnic1_prsnt_n-dbg-ctrl
cnic2_prsnt_n
cnic2_prsnt_n-dbg-ctrl
cpld_cpu1_fivr_fault
cpld_cpu1_fivr_fault-dbg-ctrl
cpld_cpu2_fivr_fault
cpld_cpu2_fivr_fault-dbg-ctrl
cpld_finish_bit
cpld_finish_bit-dbg-ctrl
cpld_rev_id_x
cpld_rev_id_x-dbg-ctrl
cpld_rev_id_y
cpld_rev_id_y-dbg-ctrl
cpld_rev_id_z
cpld_rev_id_z-dbg-ctrl
cpu2_disable
fan1_prsnt_n
fan1_prsnt_n-dbg-ctrl
fan2_prsnt_n
fan2_prsnt_n-dbg-ctrl
fan3_prsnt_n
fan3_prsnt_n-dbg-ctrl
fan4_prsnt_n
fan4_prsnt_n-dbg-ctrl
fan5_prsnt_n
fan5_prsnt_n-dbg-ctrl
fan6_prsnt_n
fan6_prsnt_n-dbg-ctrl
fan7_prsnt_n
fan7_prsnt_n-dbg-ctrl
flex_lom_pcie_bw_n
flex_lom_pcie_bw_n-dbg-ctrl
flex_lom_reset
fm_bios_post_cmplt
fm_bios_post_cmplt-dbg-ctrl
fm_bios_post_cmplt_interrupt
fm_bios_recover_boot
fm_bios_recover_boot-dbg-ctrl
fm_bmc_clr_rst_latch_r
fm_bmc_clr_rst_latch_r1
fm_bmc_x550_sdp0_1
fm_bmc_x550_sdp0_2
fm_bmc_x550_sdp0_3
fm_bmc_x550_sdp1_1
fm_clk_buf_smb_wrtlock
fm_cpld_lock_n
fm_cpu1_mem01_memhot_n
fm_cpu1_mem01_memhot_n-dbg-ctrl
fm_cpu1_mem23_memhot_n
fm_cpu1_mem23_memhot_n-dbg-ctrl
fm_cpu1_pkgid_0
fm_cpu1_pkgid_0-dbg-ctrl
fm_cpu1_pkgid_1
fm_cpu1_pkgid_1-dbg-ctrl
fm_cpu1_pkgid_2
fm_cpu1_pkgid_2-dbg-ctrl
fm_cpu1_proc_id_0
fm_cpu1_proc_id_0-dbg-ctrl
fm_cpu1_proc_id_1
fm_cpu1_proc_id_1-dbg-ctrl
fm_cpu1_prochot_n
fm_cpu1_prochot_n-dbg-ctrl
fm_cpu1_sktocc_n
fm_cpu1_sktocc_n-dbg-ctrl
fm_cpu1_sm_wp
fm_cpu1_thermtrip_latch_lvc3_n
fm_cpu1_thermtrip_latch_lvc3_n-dbg-ctrl
fm_cpu2_mem01_memhot_n
fm_cpu2_mem01_memhot_n-dbg-ctrl
fm_cpu2_mem23_memhot_n
fm_cpu2_mem23_memhot_n-dbg-ctrl
fm_cpu2_pkgid_0
fm_cpu2_pkgid_0-dbg-ctrl
fm_cpu2_pkgid_1
fm_cpu2_pkgid_1-dbg-ctrl
fm_cpu2_pkgid_2
fm_cpu2_pkgid_2-dbg-ctrl
fm_cpu2_proc_id_0
fm_cpu2_proc_id_0-dbg-ctrl
fm_cpu2_proc_id_1
fm_cpu2_proc_id_1-dbg-ctrl
fm_cpu2_prochot_n
fm_cpu2_prochot_n-dbg-ctrl
fm_cpu2_sktocc_n
fm_cpu2_sktocc_n-dbg-ctrl
fm_cpu2_sm_wp
fm_cpu2_thermtrip_latch_lvc3_n
fm_cpu2_thermtrip_latch_lvc3_n-dbg-ctrl
fm_dimma1_event_lvt3_n
fm_dimma1_event_lvt3_n-dbg-ctrl
fm_dimma2_event_lvt3_n
fm_dimma2_event_lvt3_n-dbg-ctrl
fm_dimmb1_event_lvt3_n
fm_dimmb1_event_lvt3_n-dbg-ctrl
fm_dimmb2_event_lvt3_n
fm_dimmb2_event_lvt3_n-dbg-ctrl
fm_dimmc1_event_lvt3_n
fm_dimmc1_event_lvt3_n-dbg-ctrl
fm_dimmc2_event_lvt3_n
fm_dimmc2_event_lvt3_n-dbg-ctrl
fm_dimmd1_event_lvt3_n
fm_dimmd1_event_lvt3_n-dbg-ctrl
fm_dimmd2_event_lvt3_n
fm_dimmd2_event_lvt3_n-dbg-ctrl
fm_dimme1_event_lvt3_n
fm_dimme1_event_lvt3_n-dbg-ctrl
fm_dimme2_event_lvt3_n
fm_dimme2_event_lvt3_n-dbg-ctrl
fm_dimmf1_event_lvt3_n
fm_dimmf1_event_lvt3_n-dbg-ctrl
fm_dimmf2_event_lvt3_n
fm_dimmf2_event_lvt3_n-dbg-ctrl
fm_dimmg1_event_lvt3_n
fm_dimmg1_event_lvt3_n-dbg-ctrl
fm_dimmg2_event_lvt3_n
fm_dimmg2_event_lvt3_n-dbg-ctrl
fm_dimmh1_event_lvt3_n
fm_dimmh1_event_lvt3_n-dbg-ctrl
fm_dimmh2_event_lvt3_n
fm_dimmh2_event_lvt3_n-dbg-ctrl
fm_dimmj1_event_lvt3_n
fm_dimmj1_event_lvt3_n-dbg-ctrl
fm_dimmj2_event_lvt3_n
fm_dimmj2_event_lvt3_n-dbg-ctrl
fm_dimmk1_event_lvt3_n
fm_dimmk1_event_lvt3_n-dbg-ctrl
fm_dimmk2_event_lvt3_n
fm_dimmk2_event_lvt3_n-dbg-ctrl
fm_dimml1_event_lvt3_n
fm_dimml1_event_lvt3_n-dbg-ctrl
fm_dimml2_event_lvt3_n
fm_dimml2_event_lvt3_n-dbg-ctrl
fm_dimmm1_event_lvt3_n
fm_dimmm1_event_lvt3_n-dbg-ctrl
fm_dimmm2_event_lvt3_n
fm_dimmm2_event_lvt3_n-dbg-ctrl
fm_err2_n
fm_err2_n-dbg-ctrl
fm_fbp_present_n
fm_fbp_present_n-dbg-ctrl
fm_foc_n
fm_fp_btn_present_n
fm_fp_btn_present_n-dbg-ctrl
fm_fp_kvm_present_n
fm_fp_kvm_present_n-dbg-ctrl
fm_host_tpm_present_n
fm_host_tpm_present_n-dbg-ctrl
fm_ibmc_flash_wp_n
fm_ibmc_heartbeat
fm_ibmc_lom_pcie_rst
fm_ibmc_pch_sci_lpc_n
fm_ibmc_pch_smi_lpc_n
fm_ibmc_pmbus_recover_n
fm_ibmc_rsmrst_override_n
fm_ibmc_smbus_fanout_reset_n
fm_intruder_present_bmc_n
fm_intruder_present_bmc_n-dbg-ctrl
fm_lan0_disable
fm_lan1_disable
fm_me_rcvr
fm_mraid_ifdet_id_0
fm_mraid_ifdet_id_0-dbg-ctrl
fm_mraid_ifdet_id_1
fm_mraid_ifdet_id_1-dbg-ctrl
fm_mstg_fru_wp_n
fm_nmi_event_r_n
fm_nmi_event_r_n-dbg-ctrl
fm_p12v_lom_aux_status
fm_p12v_lom_status
fm_pch_ibmc_thermtrip_n
fm_pch_ibmc_thermtrip_n-dbg-ctrl
fm_ps1_off
fm_ps2_off
fm_psoc_alert_bmc_n
fm_psoc_alert_bmc_n-dbg-ctrl
fm_psoc_alert_n
fm_psoc_alert_n-dbg-ctrl
fm_rbp_present_n
fm_rbp_present_n-dbg-ctrl
fm_rear_hdd_present_n_0
fm_rear_hdd_present_n_0-dbg-ctrl
fm_rear_hdd_present_n_1
fm_rear_hdd_present_n_1-dbg-ctrl
fm_rear_vga_present_n
fm_rear_vga_present_n-dbg-ctrl
fm_rtcrst
fm_sata_odd_present_n
fm_sata_odd_present_n-dbg-ctrl
fm_sed_xdp_present_cpld
fm_sed_xdp_present_cpld-dbg-ctrl
fm_smi_active_n
fm_smi_active_n-dbg-ctrl
fm_video_disable_n
fm_video_disable_n-dbg-ctrl
fm_x550_flsh_sck_bmc
foc_alert_n
foc_alert_n-dbg-ctrl
fp_id_btn
fp_id_btn-dbg-ctrl
fp_id_btn_interrupt
fp_id_led
fp_video_plugin_det0_n
fp_video_plugin_det0_n-dbg-ctrl
host_bmc_com_toggle
hw_rev
hw_rev-dbg-ctrl
ibmc_arm_wdog
ibmc_bios_gpio0
ibmc_bios_gpio1
ibmc_boot_spi_wp_n_out
ibmc_jtag_pld_tdo
ibmc_jtag_pld_tdo-dbg-ctrl
ibmc_jtag_prog_en
ibmc_led_postcode_0
ibmc_led_postcode_1
ibmc_md_magn_sel1_n
ibmc_md_magn_sel2_n
ibmc_pch_pmbus_dis
ibmc_prog_jtag_tck
ibmc_prog_jtag_tdi
ibmc_prog_jtag_tms
ibmc_rst_act2_n
ibmc_tpm_gpio1
ibmc_tpm_irq_n
ibmc_tpm_irq_n-dbg-ctrl
ibmc_tpm_phy_pres
ibmcbios_sel_n
ibmcbios_sel_n-dbg-ctrl
irq_caterr_dly_ctrl_n
irq_ibmc_pch_nmi_stby_n
irq_lvc3_cpu_caterr_n
irq_lvc3_cpu_caterr_n-dbg-ctrl
irq_sml0_alert_r_n
irq_sml0_alert_r_n-dbg-ctrl
lan_reset_block_n
lom_power_good_reset
mc_manufacture_mode_n
mc_manufacture_mode_n-dbg-ctrl
mem_dbg_bus_gnt_n
mem_dbg_bus_gnt_n-dbg-ctrl
mem_dbg_bus_req_n
mini_storage_id0
mini_storage_id0-dbg-ctrl
mini_storage_id1
mini_storage_id1-dbg-ctrl
mini_storage_prsnt
mini_storage_prsnt-dbg-ctrl
mlom_prsnt_n
mlom_prsnt_n-dbg-ctrl
old_ibmc_bios_sel_n
post
prog_cpld_fpga_sel_n
ps1_ac_ok
ps1_ac_ok-dbg-ctrl
ps1_prsnt
ps1_prsnt-dbg-ctrl
ps1_pwrgd
ps1_pwrgd-dbg-ctrl
ps2_ac_ok
ps2_ac_ok-dbg-ctrl
ps2_prsnt
ps2_prsnt-dbg-ctrl
ps2_pwrgd
ps2_pwrgd-dbg-ctrl
psu_pmbus_alert_n
psu_pmbus_alert_n-dbg-ctrl
pwrgd_psu_n
pwrgd_psu_n-dbg-ctrl
pwrgd_sys_pwrok
pwrgd_sys_pwrok-dbg-ctrl
rear_hdbp_prst_n
rear_hdbp_prst_n-dbg-ctrl
riser1_id_0
riser1_id_0-dbg-ctrl
riser1_id_1
riser1_id_1-dbg-ctrl
riser1_present_1_n
riser1_present_1_n-dbg-ctrl
riser1_present_2_n
riser1_present_2_n-dbg-ctrl
riser1_present_3_n
riser1_present_3_n-dbg-ctrl
riser1_present_n
riser1_present_n-dbg-ctrl
riser2_id_0
riser2_id_0-dbg-ctrl
riser2_id_1
riser2_id_1-dbg-ctrl
riser2_present_1_n
riser2_present_1_n-dbg-ctrl
riser2_present_2_n
riser2_present_2_n-dbg-ctrl
riser2_present_3_n
riser2_present_3_n-dbg-ctrl
riser2_present_n
riser2_present_n-dbg-ctrl
rsmrst_ibmc_88e6176_n
rst_bmc_fp_exp_n
rst_lvc3_cpu_reset_n
rst_lvc3_cpu_reset_n-dbg-ctrl
rst_pltrst_n_falling
rst_pltrst_n_falling-dbg-ctrl
rst_pltrst_n_rising
rst_pltrst_n_rising-dbg-ctrl
rst_riser2_smb_n
sas_hba_prsnt_n
sas_hba_prsnt_n-dbg-ctrl
sata_int_present_n
sata_int_present_n-dbg-ctrl
sdcont_ibmc_id_n
sdcont_ibmc_id_n-dbg-ctrl
sdhc_not_led
sed_cpu_debug_en_n
sed_cpu_prdy_n
sed_cpu_prdy_n-dbg-ctrl
sed_cpu_preq_n
sed_ev_cpu_pwr_debug_n
sgpio_output_en_n
smb3_pch_lockout
smbus_bmc_pch_ctrl_en
spd_mux_override
temp_warn_n
temp_warn_n-dbg-ctrl
usb_fp_en_n
usb_mux_cntrl
vrd_cpu1_vrhot_n
vrd_cpu1_vrhot_n-dbg-ctrl
vrd_cpu2_vrhot_n
vrd_cpu2_vrhot_n-dbg-ctrl
vrd_pvddq_cpu1_vrhot_n
vrd_pvddq_cpu1_vrhot_n-dbg-ctrl
vrd_pvddq_cpu2_vrhot_n
vrd_pvddq_cpu2_vrhot_n-dbg-ctrl
watchdog_reset_r_en
xdp_present_n
xdp_present_n-dbg-ctrlAfter spending a bit of time looking at these, I found a couple that are quite interesting.
/proc/cisco/gpio/cpu2_disable – This allows for the second CPU socket to be disabled. From testing (simply writing 1 to the file), this actually works and after rebooting, the server BIOS and OS only see a single socket. According to power monitoring, this also reduces server power consumption. What’s nice is that it can easily be re-enabled with just a reboot – neat! Whether this an undocumented feature or something used for debugging, I’m not sure but I have been running with a single socket now for several months and not had a single issue.
/proc/cisco/main_public_rollover_key
/proc/cisco/challenge-counter-binary
/proc/cisco/bmc-unique-id
These appear to be crytographic keys used to authenticate Cisco technical support people. Further investigation is required here.
/proc/cisco/mmio_address
/proc/cisco/mmio_memory
/proc/cisco/mmio_size
These are quite interesting and believe to be related to IMC to OS communication. They contain the size, address and possibly the contents of the memory-mapped I/O data. Memory-Mapped I/O (MMIO) is a method used for communication between the operating system and the BMC/IPMI in servers. It works by mapping BMC hardware controls into the system’s memory space, allowing the OS to interact with the BMC as if it were regular memory. This setup provides quick and direct access to the BMC for tasks like monitoring server health and managing server operations.
There is a lot more to investigate here, lots of potentially interesting devices exist in /sys and /dev too. Expect a follow up post on this.
We have fan control!
After some digging around on the IMC, I discovered a multitude of tools. One patricuralty useful one is /usr/local/bin/rack_fan_control which allows you to take full control of the server fan speeds!
[root@C240:~]$rack_fan_control -c set_override enable 20 20 20 20 20 20
Waiting for server response...
RFC command parsed and executed successfully!
[root@C240:~]$rack_fan_control -c get_fan_speeds
Waiting for server response...
Fan 1 speed: 5050 RPM
Fan 2 speed: 5194 RPM
Fan 3 speed: 5151 RPM
Fan 4 speed: 5194 RPM
Fan 5 speed: 5151 RPM
Fan 6 speed: 5194 RPM
Fan 7 speed: 5252 RPM
Fan 8 speed: 5292 RPM
Fan 9 speed: 5252 RPM
Fan 10 speed: 5292 RPM
Fan 11 speed: 5151 RPM
Fan 12 speed: 5292 RPM
RFC command parsed and executed successfully!The above command has set the fans at pulse width modulation at 30% duty cycle and silence esues! My expensive mistake has now gone away!
Unfortunatly this has a few limitations, the first one being that a reboot of IMC causes the fans to ramp back up to high speeds again.
Can we make this persistent?
The following is an excerpt from the mount command which shows that there is a non-volatile filesystem mount on the OS. It is very likely that a carefully crafted script could be used to establish persistence.
/dev/ubi0_0 on /mnt/nand type ubifs (rw,sync,relatime,no_chk_data_crc,assert=read-only,ubi=0,vol=0)In theory, a script placed on this mount could be executed at startup.
I will conclude this blog post now and will release a more detailed technical rundown once the CVE has been issued for the vulnerability.
Leave a Reply