Windows Support

[Paddy Horan]

Name says it all...

I'm trying to pull together a PR.

Compiling on windows fails on the call to posix_memalign, obviously not available on windows. Checking the arrow C++ source I should be able to use _aligned_malloc. However, this is not exposed by libc.

Definitely a little out of my depth trying to add _aligned_malloc to libc. Hoping maybe someone watching this might be able to comment on the libc issue above and get me on my way...

[Jihoon Son]

Well, I googled to find a library for _aligned_malloc, but could find any. If it doesn't exist, you can simply make a wrapper function and use it when the platform is windows. Please check https://doc.rust-lang.org/book/first-edition/ffi.html and https://stackoverflow.com/questions/43292357/detect-platform-in-rust.

[Paddy Horan]

Thanks @jihoonson, I'll take a look.

[Paddy Horan]

@jihoonson @andygrove I have windows support working (if I comment out the posix version, see below). However, I'd like your input on something.

There were a few places where memory_pool.rs had to be changed. One of those is allocate_aligned which uses posix_memaligned. I can use conditional compilation to use the windows equivalent. Here is how I would like allocate_aligned to work:

fn allocate_aligned(size: i64) -> Result<*const u8, ArrowError> {
  if cfg!(windows) {
    let page = unsafe { _aligned_malloc(size as libc::size_t,
                                        ALIGNMENT as libc::size_t) };
    match page {
      0 => panic!("Unable to allocate {} bytes", size),
      _ => Ok(unsafe {mem::transmute::<libc::size_t, *const u8>(page)})
    }

  } else {
    unsafe {
      let mut page: *mut libc::c_void = mem::uninitialized();
      let result = libc::posix_memalign(&mut page, ALIGNMENT, size as usize);
//    println!("allocated aligned memory of {} at {:?}", size, page);
      match result {
        libc::ENOMEM => Err(ArrowError::out_of_memory(format!("malloc of size {} failed", size))),
        libc::EINVAL => Err(ArrowError::invalid(format!("invalid alignment parameter: {}", ALIGNMENT))),
        0 => Ok(mem::transmute::<*mut libc::c_void, *const u8>(page)),
        _ => panic!("unknown allocation result: {}", result)
      }
    }
  }
}

However, this does not work because libc::posix_memalign is not included in the libc namespace on windows.

I have looked at libc and they handle this kind of thing by creating separate modules and conditionally using them.

Here is what I am thinking:

• change memory_pool to be a module
• add windows.rs and posix.rs beside mod.rs in the memory_pool folder
• add allocate_aligned and free_aligned for both windows and posix with the same signatures
• conditionally use in src/memory_pool/mod.rs

Thoughts? Is there any easier way of doing this that I'm missing?

[Paddy Horan]

@jihoonson @andygrove disregard the above question, PR coming soon.

[Jihoon Son]

@paddyhoran thanks for working on this. Please take a look apache/arrow#1804 too. The native Rust implementation will be more actively developed under the apache repository.

[Paddy Horan]

Thanks @jihoonson. I actually did see that on the arrow mailing list today. I guess we should all focus our efforts there.