Skip to content
This repository has been archived by the owner on Mar 10, 2020. It is now read-only.

Add HACKING.md #83

Merged
merged 10 commits into from
Dec 4, 2016
Merged

Add HACKING.md #83

merged 10 commits into from
Dec 4, 2016

Conversation

izgzhen
Copy link
Contributor

@izgzhen izgzhen commented Nov 23, 2016 •
edited

Fixes issue #81 and #14. Not perfect yet so feedbacks are welcomed.

Now I have some more considerations.

First, HACKING.md is a dev guide, so naturally there should be a user-guide. I remember that we have some packages and even docker image, so maybe their usages will be put in the user guide. (However, I suspect that the user guide is of a lower priority.)

Second, for the related text in wiki, I plan not to delete them now, but simply add a link in them since I actually did some simplifications so older texts might still be useful in some sense. But in the long run maybe a more unified doc is better.

@abailly
Copy link

abailly commented Nov 23, 2016

I would be happy to test those HACKING instructions!

@acw
Copy link
Contributor

acw commented Nov 23, 2016

This is awesome!

@izgzhen: One small change request: can you bump Fedora 22 to Fedora 24?
@abailly: I would love someone to do a test run.

@izgzhen
Copy link
Contributor Author

izgzhen commented Nov 24, 2016

@acw No problem, just needing some time though :)

@abailly
Copy link

abailly commented Nov 24, 2016 via email

@izgzhen
Copy link
Contributor Author

izgzhen commented Nov 24, 2016

Really? .... interesting. I have set up a basic Fedora 24 installation just a few hours early as well, but have not arrived at the grub step yet. I think I can help investigate a bit later.

@abailly
Copy link

abailly commented Nov 24, 2016

Got some information from this page: https://www.digitalocean.com/community/tutorials/how-to-update-a-digitalocean-server-s-kernel#booting-to-a-non-default-kernel-on-syslinux-distributions

@abailly
Copy link

abailly commented Nov 24, 2016 •
edited

And there is no libgmp, only gmp on 23 and 24

@izgzhen
Copy link
Contributor Author

izgzhen commented Nov 24, 2016

And 24 Server's support for distro emacs is also poor ... I guess supporting 24 would not be trivial.

@izgzhen
Copy link
Contributor Author

izgzhen commented Nov 24, 2016

I am trying Fedora Workstation 24 now. Since there is xen-libs pre-installed, I have to dnf erase it (I can't boot normally when I did this in Fedora 22 Workstation so I favoured the Server bundle). Then I skipped the grub part and directly reboot into "Fedora, with Xen Hypervisor", after a long booting process, it panicked into dracut mode ...

@abailly
Copy link

abailly commented Nov 24, 2016

Busy building Halvm on a Fedora 23 server in VirtualBox. got some comments and minor deviations from HACKING.md so far, nothing major. Probably not the right place to chat but the cumbersomeness of the building process made me think it could be nice to have a way to ask stack to use HaLVM setup... I think this should be possible provided there is a way to provide pre-built binaries for custom GHC installation.

@abailly
Copy link

abailly commented Nov 25, 2016 •
edited

I can confirm I was able to build HaLVM and nearly all examples following @izgzhen 's instructions.
My settings:

  • VirtualBox 5.0.8 r103449
  • MacOS X 10.10.5
  • Installed Fedora 23 Server using DVD Iso image

Some minor comments:

  • setting up port forwarding: NIC must be in NAT mode
  • setting up ~/.ssh/config halvm section but I don't have ssh-copy-id
  • dnf install : OK if replacing libgmp with gmp
  • grub config : OK
  • xl list : OK
  • configuring network (note vim is not installed by default...)
    • name of directory typo: sudo vim /etc/sysconfig/network-scripts/ifcfg-enXXXX
    • ping galois.com : OK
    • patting myself on the back...
  • building Halvm
    • installing git: sudo dnf install git
    • installing missing packages: gmp-devel halvm-xen-devel
    • building...
    • build ok, install ok
  • building examples:
    • failed to build IVC example: missing -lcrypto, not sure which library is this

@izgzhen
Copy link
Contributor Author

izgzhen commented Nov 25, 2016

Cool! Thank you very much. Honestly speaking, I haven't tested all examples yet. I remembered that I bumped into -lcrypto problem a long time ago. I guess it is OpenSSL? (Sorry I don't have a VM around now)

And which kind of Fedora did you use? Workstation or Server or something else?

@abailly
Copy link

abailly commented Nov 25, 2016

Server. Updated initial comment :) Don't have much time to test examples right now but I am confident they will work, bar networking issues (always had troubles getting DHCP to work on Xen in VM on my Mac)

@izgzhen
Copy link
Contributor Author

izgzhen commented Nov 26, 2016

First I tried Server 24 and I found that:

  1. @abailly Looks like grub2 is still in use in 24?
  2. However, the booting still failed to dracut in Server Edition ... @acw What is your opinion on this?

@acw
Copy link
Contributor

acw commented Nov 26, 2016

Hmmm. Usually boot failures like that signal a mismatch between Xen and the Linux kernel. One thing to check is whether or not the system boots with the standard xen packages rather than the halvm-xen ones. (The HaLVM will still run on xen, it's just harder to debug broken ones.) At the same time, I'll kick off a fresh build of the halvm-xen packages, just in case we're behind a version or two.

@abailly
Copy link

abailly commented Nov 26, 2016

@izgzhen I am surprised: When I tried to work on a Fedora 24 image on DO, I could not find any trace of grub configuration. But this might be because DO provides images with a specific Fedora 24 configuration...

@izgzhen
Copy link
Contributor Author

izgzhen commented Nov 27, 2016 •
edited

virtualbox_halvm24_27_11_2016_10_44_25

This is what happened: After dnf install either halvm-xen or xen, rebooting will fail into this, at least on my machine. I can't find similar problems on the Internet.

EDIT: I tried to do the same thing on my Mac, and it is the same symptom. Maybe we should file an issue to Xen or Fedora...

@izgzhen
Copy link
Contributor Author

izgzhen commented Nov 30, 2016

My problem is fixed and I should take back my words ... we still need to dnf update before installing xen. I am currently testing HaLVM Fedora 24

@izgzhen
Copy link
Contributor Author

izgzhen commented Nov 30, 2016

Running Hello example (looks not very encouraging):

[halvm@localhost Hello]$ make run
sudo xl create Hello.config
Parsing config from Hello.config
sleep 1
sudo xl dmesg -c
 Xen 4.6.4-1.fc24
(XEN) Xen version 4.6.4 (hudson@dev.galois.com) (gcc (GCC) 6.2.1 20160916 (Red Hat 6.2.1-2)) debug=n Tue Nov 29 21:32:03 PST 2016
(XEN) Latest ChangeSet: 
(XEN) Bootloader: GRUB 2.02~beta3
(XEN) Command line: placeholder
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN) Disc information:
(XEN)  Found 1 MBR signatures
(XEN)  Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009fc00 (usable)
(XEN)  000000000009fc00 - 00000000000a0000 (reserved)
(XEN)  00000000000f0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 000000007fff0000 (usable)
(XEN)  000000007fff0000 - 0000000080000000 (ACPI data)
(XEN)  00000000fec00000 - 00000000fec01000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000fffc0000 - 0000000100000000 (reserved)
(XEN) System RAM: 2047MB (2096700kB)
(XEN) ACPI: RSDP 000E0000, 0024 (r2 VBOX  )
(XEN) ACPI: XSDT 7FFF0030, 003C (r1 VBOX   VBOXXSDT        1 ASL        61)
(XEN) ACPI: FACP 7FFF00F0, 00F4 (r4 VBOX   VBOXFACP        1 ASL        61)
(XEN) ACPI: DSDT 7FFF0470, 210F (r1 VBOX   VBOXBIOS        2 INTL 20160108)
(XEN) ACPI: FACS 7FFF0200, 0040
(XEN) ACPI: APIC 7FFF0240, 0054 (r2 VBOX   VBOXAPIC        1 ASL        61)
(XEN) ACPI: SSDT 7FFF02A0, 01CC (r1 VBOX   VBOXCPUT        2 INTL 20160108)
(XEN) Domain heap initialised
(XEN) Processor #0 7:10 APIC version 20
(XEN) IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) Not enabling x2APIC: depends on iommu_supports_eim.
(XEN) XSM Framework v1.0.0 initialized
(XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 3391.587 MHz processor.
(XEN) Initing memory sharing.
(XEN) I/O virtualisation disabled
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) Platform timer is 3.579MHz ACPI PM Timer
(XEN) Allocated console ring of 16 KiB.
(XEN) Brought up 1 CPUs
(XEN) CPUIDLE: disabled due to no HPET. Force enable with 'cpuidle'.
(XEN) xenoprof: Initialization failed. Intel processor family 6 model 58is not supported
(XEN) Dom0 has maximum 208 PIRQs
(XEN) *** LOADING DOMAIN 0 ***
(XEN) elf_parse_binary: phdr: paddr=0x1000000 memsz=0xd43000
(XEN) elf_parse_binary: phdr: paddr=0x1e00000 memsz=0x164000
(XEN) elf_parse_binary: phdr: paddr=0x1f64000 memsz=0x1a8d8
(XEN) elf_parse_binary: phdr: paddr=0x1f7f000 memsz=0x329000
(XEN) elf_parse_binary: memory: 0x1000000 -> 0x22a8000
(XEN) elf_xen_parse_note: GUEST_OS = "linux"
(XEN) elf_xen_parse_note: GUEST_VERSION = "2.6"
(XEN) elf_xen_parse_note: XEN_VERSION = "xen-3.0"
(XEN) elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) elf_xen_parse_note: INIT_P2M = 0x8000000000
(XEN) elf_xen_parse_note: ENTRY = 0xffffffff81f7f180
(XEN) elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000
(XEN) elf_xen_parse_note: FEATURES = "!writable_page_tables|pae_pgdir_above_4gb|writable_descriptor_tables|auto_translated_physmap|supervisor_mode_kernel"
(XEN) elf_xen_parse_note: SUPPORTED_FEATURES = 0x90d
(XEN) elf_xen_parse_note: PAE_MODE = "yes"
(XEN) elf_xen_parse_note: LOADER = "generic"
(XEN) elf_xen_parse_note: unknown xen elf note (0xd)
(XEN) elf_xen_parse_note: SUSPEND_CANCEL = 0x1
(XEN) elf_xen_parse_note: MOD_START_PFN = 0x1
(XEN) elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
(XEN) elf_xen_parse_note: PADDR_OFFSET = 0x0
(XEN) elf_xen_addr_calc_check: addresses:
(XEN)     virt_base        = 0xffffffff80000000
(XEN)     elf_paddr_offset = 0x0
(XEN)     virt_offset      = 0xffffffff80000000
(XEN)     virt_kstart      = 0xffffffff81000000
(XEN)     virt_kend        = 0xffffffff822a8000
(XEN)     virt_entry       = 0xffffffff81f7f180
(XEN)     p2m_base         = 0x8000000000
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x22a8000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000074000000->0000000078000000 (464083 pages to be allocated)
(XEN)  Init. ramdisk: 000000007e6ce000->000000007f9ffa84
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff822a8000
(XEN)  Init. ramdisk: 0000000000000000->0000000000000000
(XEN)  Phys-Mach map: 0000008000000000->00000080003b4028
(XEN)  Start info:    ffffffff822a8000->ffffffff822a84b4
(XEN)  Page tables:   ffffffff822a9000->ffffffff822be000
(XEN)  Boot stack:    ffffffff822be000->ffffffff822bf000
(XEN)  TOTAL:         ffffffff80000000->ffffffff82400000
(XEN)  ENTRY ADDRESS: ffffffff81f7f180
(XEN) Dom0 has maximum 1 VCPUs
(XEN) elf_load_binary: phdr 0 at 0xffffffff81000000 -> 0xffffffff81d43000
(XEN) elf_load_binary: phdr 1 at 0xffffffff81e00000 -> 0xffffffff81f64000
(XEN) elf_load_binary: phdr 2 at 0xffffffff81f64000 -> 0xffffffff81f7e8d8
(XEN) elf_load_binary: phdr 3 at 0xffffffff81f7f000 -> 0xffffffff820ff000
(XEN) Scrubbing Free RAM on 1 nodes using 1 CPUs
(XEN) ................done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 288kB init memory.
(XEN) d0 attempted to change d0v0's CR4 flags 00000620 -> 00040660

@izgzhen
Copy link
Contributor Author

izgzhen commented Dec 1, 2016

By the way, building GHC on Fedora 24 required libtinfo.so.5, but there is only a newer one: libtinfo.so.6, so I actually did something bad sudo ln -s /usr/lib64/libtinfo.so.6 /usr/lib64/libtinfo.so.5

@acw
Copy link
Contributor

acw commented Dec 1, 2016

Yes, I noticed the same thing (libtinfo.so.6 -> libtinfo.so.5 starting to try to get the HaLVM to build on Fedora 25. It's dirty and ugly, but it works.

Given that 25 just got released, getting this running on 24 just became much more important. So let's see if we can figure this out!

With regard to the Hello World example, was this with 'halvm-xen' and with "console_to_ring loglvl=all guest_loglvl=all" on the command line? I always forget the latter.

@izgzhen
Copy link
Contributor Author

izgzhen commented Dec 2, 2016

You are right 😂 I skipped the grub part. So, for libtinfo problem, I guess either we find a older version of it working on Fedora 24, or upgrading GHC (not sure if it will work). For now I will add this hack to instructions.

@izgzhen
Copy link
Contributor Author

izgzhen commented Dec 2, 2016

Everything except for WebServer in examples are tested, and now:

In Core examples, DomainInfo doesn't work (which is already complained in #47 and #52 :P):

sudo xl create DomainInfo.config -c
Parsing config from DomainInfo.config
libxl: error: libxl_dom.c:37:libxl__domain_type: unable to get domain type for domid=7
xl: unable to exec console client: No such file or directory
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: console child [1782] exited with error status 1

And a seemly innocent problem:

[halvm@localhost PacketCount]$ make run
sudo xl destroy PacketCount
PacketCount is an invalid domain identifier (rc=-6)
Makefile:11: recipe for target 'run' failed
make: [run] Error 2 (ignored)
sudo xl create PacketCount.config -c
Failed to read config file: PacketCount.config: No such file or directory
Makefile:11: recipe for target 'run' failed
make: *** [run] Error 3
```

@izgzhen
Copy link
Contributor Author

izgzhen commented Dec 3, 2016 •
edited

Note: when I compiled HaNS (on Fedora 24), there will be an error indicating it can't find libHSprocess-1.2.0.0-ghc7.8.4.so. export LD_LIBRARY_PATH=/usr/local/lib/HaLVM-2.2.1/lib will solve it. I don't know if something went wrong in my build or it is a Fedora 24 peculiarity.

@izgzhen
Copy link
Contributor Author

izgzhen commented Dec 3, 2016 •
edited

I think we can put WebServer problem aside now .... I tried to update WebServer.hs (see https://gist.github.com/izgzhen/446e8cd8a076193dfbbd488ab9df75c7, it doesn't work very well), and also tried to track the older dependencies of the network stack (but it is a mess :(, see #85, #87).

So I think it is ready for merge now.

@acw acw merged commit 8d61fe7 into GaloisInc:master Dec 4, 2016
@izgzhen izgzhen mentioned this pull request Dec 7, 2016
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@izgzhen @abailly @acw