Unable to use smack labels(xattr) with v3.19 aufs

[Arun Chandran]

Hi All,

I am using branch 'origin/aufs3.19' from
git://github.com/sfjro/aufs3-linux.git. and want to use smack with
aufs. So I enabled the option CONFIG_AUFS_XATTR and booted the kernel.

To test it I did

1) mount /dev/hdb1 to /mnt

# mount | grep hdb1
/dev/hdb1 on /mnt type ext4 (rw,relatime,data=ordered)

2) Used the below test script to make the aufs mount

----------------------------------------
# pwd
/mnt
# cat test_aufs_cp.sh
mkdir a b c
chsmack -a 'k1' a
chsmack -a 'k1' b
chsmack -a 'k1' c

cd a
for i in 1 2 3
do
echo $i > a$i.txt
chsmack -a 'k1' a$i.txt
done

cd ../b
for i in 1 2 3
do
echo $i > b$i.txt
chsmack -a 'k1' b$i.txt
done

cd ../c
for i in 1 2 3
do
echo $i > c$i.txt
chsmack -a 'k1' c$i.txt
done

cd ..

mkdir mnt

mount -t aufs -o
br=./a=rw:./b=ro:./c=ro:,noplink,smackfsdef=k1,smackfsroot=k1 -o
udba=reval none ./mnt
---------------------------------

3) Run the script and make sure that it is mounted
# mount | grep aufs
none on /mnt/mnt type aufs (rw,relatime,si=669ecc04df7a829e,noplink)

4) Now if I cd to /mnt/mnt and do 'ls' it hangs and I get the below oops.


# dmesg
[ 148.855382] ------------[ cut here ]------------
[ 148.855382] kernel BUG at fs/aufs/sbinfo.c:336!
[ 148.855382] invalid opcode: 0000 [#1] PREEMPT SMP
[ 148.855382] Modules linked in:
[ 148.855382] CPU: 0 PID: 140 Comm: ls Not tainted 3.19.0-yocto-standard+ #1
[ 148.855382] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Bochs 01/01/2011
[ 148.855382] task: ffff880006749800 ti: ffff880000078000 task.ti:
ffff880000078000
[ 148.855382] RIP: 0010:[<ffffffff813b8598>] [<ffffffff813b8598>]
si_pid_set+0x78/0xd0
[ 148.855382] RSP: 0018:ffff88000007b9e8 EFLAGS: 00000286
[ 148.855382] RAX: ffff88000669c000 RBX: 000000000000008b RCX: 0000000000000100
[ 148.855382] RDX: 00000000ffffffff RSI: 0000000000000028 RDI: ffff88000669b000
[ 148.855382] RBP: ffff88000007ba08 R08: 0000000000000000 R09: 0000000000000000
[ 148.855382] R10: ffff880006740d00 R11: 0000000000000000 R12: ffff88000668b800
[ 148.855382] R13: ffff88000669b000 R14: 0000000000000028 R15: ffff88000672e480
[ 148.855382] FS: 0000000000000000(0000) GS:ffff880007c00000(0063)
knlGS:00000000f75f9700
[ 148.855382] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 148.855382] CR2: 00000000f774f0d3 CR3: 0000000006757000 CR4: 00000000000006f0
[ 148.855382] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 148.855382] DR3: 0000000000000000 DR6: 0000000000000000 DR7: 0000000000000000
[ 148.855382] Stack:
[ 148.855382] ffff88000007b9f8 ffff88000669b000 ffff88000668b800
ffff8800069fc300
[ 148.855382] ffff88000007ba78 ffffffff813b8685 0000000000000000
0000000000000000
[ 148.855382] ffff88000007ba48 ffffffff81968b68 ffffffff811a0300
ffff8800066b3a00
[ 148.855382] Call Trace:
[ 148.855382] [<ffffffff813b8685>] si_read_lock+0x95/0x170
[ 148.855382] [<ffffffff81968b68>] ? preempt_schedule_irq+0x48/0x80
[ 148.855382] [<ffffffff811a0300>] ? inode_init_always+0xf0/0x1e0
[ 148.855382] [<ffffffff813bf52e>] ? au_xino_do_write+0x3e/0x100
[ 148.855382] [<ffffffff813ecfcb>] au_lgxattr+0x2b/0x1e0
[ 148.855382] [<ffffffff81177ec6>] ? kmem_cache_alloc_trace+0x136/0x250
[ 148.855382] [<ffffffff813ed7a9>] aufs_getxattr+0x29/0x30
[ 148.855382] [<ffffffff81408d8d>] smk_fetch.isra.21+0x5d/0xa0
[ 148.855382] [<ffffffff81408f27>] smack_d_instantiate+0x157/0x330
[ 148.855382] [<ffffffff8196bd4b>] ? _raw_spin_lock+0x1b/0x60
[ 148.855382] [<ffffffff814030eb>] security_d_instantiate+0x1b/0x30
[ 148.855382] [<ffffffff8119da1a>] d_splice_alias+0x9a/0x2f0
[ 148.855382] [<ffffffff813ddbb4>] aufs_lookup+0x244/0x430
[ 148.855382] [<ffffffff811903bd>] lookup_real+0x1d/0x70
[ 148.855382] [<ffffffff81191123>] __lookup_hash+0x33/0x40
[ 148.855382] [<ffffffff81192dc3>] lookup_slow+0x43/0xc0
[ 148.855382] [<ffffffff81195692>] path_lookupat+0x652/0x7b0
[ 148.855382] [<ffffffff81177c1e>] ? kmem_cache_alloc+0xde/0x250
[ 148.855382] [<ffffffff81177bae>] ? kmem_cache_alloc+0x6e/0x250
[ 148.855382] [<ffffffff81196770>] ? getname_flags+0x30/0x130
[ 148.855382] [<ffffffff81195816>] filename_lookup.isra.32+0x26/0x80
[ 148.855382] [<ffffffff811970f4>] user_path_at_empty+0x54/0xa0
[ 148.855382] [<ffffffff81968039>] ? __schedule+0x3f9/0xa40
[ 148.855382] [<ffffffff81197151>] user_path_at+0x11/0x20
[ 148.855382] [<ffffffff8118b872>] vfs_fstatat+0x52/0xa0
[ 148.855382] [<ffffffff8118b8fe>] vfs_lstat+0x1e/0x20
[ 148.855382] [<ffffffff8104f8ba>] sys32_lstat64+0x1a/0x40
[ 148.855382] [<ffffffff81003d23>] ? do_device_not_available+0x23/0x30
[ 148.855382] [<ffffffff8196e25e>] ? device_not_available+0x1e/0x30
[ 148.855382] [<ffffffff8196eea8>] cstar_dispatch+0x7/0x25
[ 148.855382] Code: 11 01 d3 81 e3 ff 7f 00 00 29 d3 48 63 db 48 0f
a3 18 19 d2 85 d2 75 13 3e 48 0f ab 18 48 83 c4 08 5b 41 5c 41 5d 5d
c3 0f 1f 00 <0f> 0b 66 0f 1f 44 00 00 4c 8d a7 50 04 00 00 4c 89 e7 e8
f1 1b
[ 148.855382] RIP [<ffffffff813b8598>] si_pid_set+0x78/0xd0
[ 148.855382] RSP <ffff88000007b9e8>
[ 148.865550] ---[ end trace e932f7d7f14c8e88 ]---
[ 148.866532] ------------[ cut here ]------------
[ 148.866599] kernel BUG at fs/aufs/sbinfo.c:336!
[ 148.866641] invalid opcode: 0000 [#2] PREEMPT SMP
[ 148.866712] Modules linked in:
[ 148.866818] CPU: 0 PID: 140 Comm: ls Tainted: G D
3.19.0-yocto-standard+ #1
[ 148.866868] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Bochs 01/01/2011
[ 148.866968] task: ffff880006749800 ti: ffff880000078000 task.ti:
ffff880000078000
[ 148.867019] RIP: 0010:[<ffffffff813b8598>] [<ffffffff813b8598>]
si_pid_set+0x78/0xd0
[ 148.867112] RSP: 0018:ffff88000007b5f8 EFLAGS: 00000286
[ 148.867161] RAX: ffff88000669c000 RBX: 000000000000008b RCX: ffff880006689000
[ 148.867165] RDX: 00000000ffffffff RSI: ffff8800070cb400 RDI: ffff88000669b000
[ 148.867165] RBP: ffff88000007b618 R08: ffff880007ffd000 R09: 0000000000000007
[ 148.867165] R10: ffffffff81ee2808 R11: 0000000000000000 R12: ffff88000668b800
[ 148.867165] R13: ffff88000669b000 R14: ffff880006740700 R15: ffff88000669b000
[ 148.867165] FS: 0000000000000000(0000) GS:ffff880007c00000(0000)
knlGS:0000000000000000
[ 148.867165] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 148.867165] CR2: 00000000f774f0d3 CR3: 0000000001e11000 CR4: 00000000000006f0
[ 148.867165] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 148.867165] DR3: 0000000000000000 DR6: 0000000000000000 DR7: 0000000000000000
[ 148.867165] Stack:
[ 148.867165] ffff88000007b618 ffff8800069f8060 ffff88000668b800
ffffffff813d8d70
[ 148.867165] ffff88000007b668 ffffffff813d5426 ffff88000007b658
ffff8800070cb400
[ 148.867165] 0000000000000051 ffff880006740700 0000000000000000
ffff8800070cb400
[ 148.867165] Call Trace:
[ 148.867165] [<ffffffff813d8d70>] ? aufs_flush_dir+0x20/0x20
[ 148.867165] [<ffffffff813d5426>] au_do_flush+0x46/0x260
[ 148.867165] [<ffffffff813d8d65>] aufs_flush_dir+0x15/0x20
[ 148.867165] [<ffffffff81184ad3>] filp_close+0x33/0x80
[ 148.867165] [<ffffffff811a3e98>] put_files_struct+0x78/0xd0
[ 148.867165] [<ffffffff811a3f9b>] exit_files+0x4b/0x60
[ 148.867165] [<ffffffff81056ecb>] do_exit+0x57b/0xb70
[ 148.867165] [<ffffffff8109cba5>] ? kmsg_dump+0xc5/0xd0
[ 148.867165] [<ffffffff8100692e>] oops_end+0x7e/0xb0
[ 148.867165] [<ffffffff81006bcb>] die+0x4b/0x70
[ 148.867165] [<ffffffff810030fa>] do_trap+0xba/0x150
[ 148.867165] [<ffffffff8100336a>] do_error_trap+0x7a/0xf0
[ 148.867165] [<ffffffff813b8598>] ? si_pid_set+0x78/0xd0
[ 148.867165] [<ffffffff814639f7>] ? debug_smp_processor_id+0x17/0x20
[ 148.867165] [<ffffffff81186460>] ? new_sync_read+0xb0/0xb0
[ 148.867165] [<ffffffff81003da0>] do_invalid_op+0x20/0x30
[ 148.867165] [<ffffffff8196e22e>] invalid_op+0x1e/0x30
[ 148.867165] [<ffffffff813b8598>] ? si_pid_set+0x78/0xd0
[ 148.867165] [<ffffffff813b8685>] si_read_lock+0x95/0x170
[ 148.867165] [<ffffffff81968b68>] ? preempt_schedule_irq+0x48/0x80
[ 148.867165] [<ffffffff811a0300>] ? inode_init_always+0xf0/0x1e0
[ 148.867165] [<ffffffff813bf52e>] ? au_xino_do_write+0x3e/0x100
[ 148.867165] [<ffffffff813ecfcb>] au_lgxattr+0x2b/0x1e0
[ 148.867165] [<ffffffff81177ec6>] ? kmem_cache_alloc_trace+0x136/0x250
[ 148.867165] [<ffffffff813ed7a9>] aufs_getxattr+0x29/0x30
[ 148.867165] [<ffffffff81408d8d>] smk_fetch.isra.21+0x5d/0xa0
[ 148.867165] [<ffffffff81408f27>] smack_d_instantiate+0x157/0x330
[ 148.867165] [<ffffffff8196bd4b>] ? _raw_spin_lock+0x1b/0x60
[ 148.867165] [<ffffffff814030eb>] security_d_instantiate+0x1b/0x30
[ 148.867165] [<ffffffff8119da1a>] d_splice_alias+0x9a/0x2f0
[ 148.867165] [<ffffffff813ddbb4>] aufs_lookup+0x244/0x430
[ 148.867165] [<ffffffff811903bd>] lookup_real+0x1d/0x70
[ 148.867165] [<ffffffff81191123>] __lookup_hash+0x33/0x40
[ 148.867165] [<ffffffff81192dc3>] lookup_slow+0x43/0xc0
[ 148.867165] [<ffffffff81195692>] path_lookupat+0x652/0x7b0
[ 148.867165] [<ffffffff81177c1e>] ? kmem_cache_alloc+0xde/0x250
[ 148.867165] [<ffffffff81177bae>] ? kmem_cache_alloc+0x6e/0x250
[ 148.867165] [<ffffffff81196770>] ? getname_flags+0x30/0x130
[ 148.867165] [<ffffffff81195816>] filename_lookup.isra.32+0x26/0x80
[ 148.867165] [<ffffffff811970f4>] user_path_at_empty+0x54/0xa0
[ 148.867165] [<ffffffff81968039>] ? __schedule+0x3f9/0xa40
[ 148.867165] [<ffffffff81197151>] user_path_at+0x11/0x20
[ 148.867165] [<ffffffff8118b872>] vfs_fstatat+0x52/0xa0
[ 148.867165] [<ffffffff8118b8fe>] vfs_lstat+0x1e/0x20
[ 148.867165] [<ffffffff8104f8ba>] sys32_lstat64+0x1a/0x40
[ 148.867165] [<ffffffff81003d23>] ? do_device_not_available+0x23/0x30
[ 148.867165] [<ffffffff8196e25e>] ? device_not_available+0x1e/0x30
[ 148.867165] [<ffffffff8196eea8>] cstar_dispatch+0x7/0x25
[ 148.867165] Code: 11 01 d3 81 e3 ff 7f 00 00 29 d3 48 63 db 48 0f
a3 18 19 d2 85 d2 75 13 3e 48 0f ab 18 48 83 c4 08 5b 41 5c 41 5d 5d
c3 0f 1f 00 <0f> 0b 66 0f 1f 44 00 00 4c 8d a7 50 04 00 00 4c 89 e7 e8
f1 1b
[ 148.867165] RIP [<ffffffff813b8598>] si_pid_set+0x78/0xd0
[ 148.867165] RSP <ffff88000007b5f8>
[ 148.879960] ---[ end trace e932f7d7f14c8e89 ]---
[ 148.880278] Fixing recursive fault but reboot is needed!
#


5) The aufs and ext4 options enabled in the kernel are

CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
CONFIG_AUFS_FS=y
CONFIG_AUFS_BRANCH_MAX_127=y
CONFIG_AUFS_SBILIST=y
CONFIG_AUFS_XATTR=y
CONFIG_AUFS_BDEV_LOOP=y
CONFIG_AUFS_DEBUG=y
CONFIG_AUFS_MAGIC_SYSRQ=y

Is this the correct way to use xattr with aufs? or v3.19 does not
support all the xattr(like the smack security label)?

--Arun

[s***@users.sourceforge.net]

Hello Arun,

Arun Chandran:
> 4) Now if I cd to /mnt/mnt and do 'ls' it hangs and I get the below oops.
>
>
> # dmesg
> [ 148.855382] ------------[ cut here ]------------
> [ 148.855382] kernel BUG at fs/aufs/sbinfo.c:336!
:::

That is interesting.
Smack enters aufs twice.

Generally a systemcall enters VFS and then aufs. Once it enters aufs,
aufs calls si_read_lock() which prohibits the branch manipulation
(add/del branches in aufs) during the systemcall. Leaving aufs,
si_read_unlock() is called and the branch manipulataion is
allowed. Finally VFS and the systemcall return to the userspace.

According to your stack trace,
- lstat(2) is issued
- VFS tries resolving the given path, and calls aufs_lookup().
- aufs finds the file (probably), and calls d_instatiate() which caches
the filepath in memory.
- d_instantiate() calls smack_d_instantiate(). obviously it should
happen when smack is enabled only.
- smack_d_instantiate() wants a specific XATTR set to that file and
calls aufs_getxattr().
What a surprise! aufs_getxattr() should be called via systemcall as
getxattr(2). The process should not re-enter aufs before completing
aufs_lookup(). At least, aufs doesn't expect such behaviour. And aufs
detects this "re-entering" behaviour and produces this message. If you
had not enabled CONFIG_AUFS_DEBUG, the message would not be
produced. But the behaviour would not be reliable anymore, I am
afraid. It was good to enable CONFIG_AUFS_DEBUG.

I don't know why smack people designed such behaviour. That is
inetresting. At the same time, I think it ugly. Why didn't they create a
hook at the very end of lookup, I wonder.

Anyway here is a quick hack for you. Won't you test it?


J. R. Okajima

[Arun Chandran]

On Sat, Jan 7, 2017 at 2:15 AM, <***@users.sourceforge.net> wrote:
> Hello Arun,
>
> Arun Chandran:
>> 4) Now if I cd to /mnt/mnt and do 'ls' it hangs and I get the below oops.
>>
>>
>> # dmesg
>> [ 148.855382] ------------[ cut here ]------------
>> [ 148.855382] kernel BUG at fs/aufs/sbinfo.c:336!
> :::
>
> That is interesting.
> Smack enters aufs twice.
>
> Generally a systemcall enters VFS and then aufs. Once it enters aufs,
> aufs calls si_read_lock() which prohibits the branch manipulation
> (add/del branches in aufs) during the systemcall. Leaving aufs,
> si_read_unlock() is called and the branch manipulataion is
> allowed. Finally VFS and the systemcall return to the userspace.
>
> According to your stack trace,
> - lstat(2) is issued
> - VFS tries resolving the given path, and calls aufs_lookup().
> - aufs finds the file (probably), and calls d_instatiate() which caches
> the filepath in memory.
> - d_instantiate() calls smack_d_instantiate(). obviously it should
> happen when smack is enabled only.
> - smack_d_instantiate() wants a specific XATTR set to that file and
> calls aufs_getxattr().
> What a surprise! aufs_getxattr() should be called via systemcall as
> getxattr(2). The process should not re-enter aufs before completing
> aufs_lookup(). At least, aufs doesn't expect such behaviour. And aufs
> detects this "re-entering" behaviour and produces this message. If you
> had not enabled CONFIG_AUFS_DEBUG, the message would not be
> produced. But the behaviour would not be reliable anymore, I am
> afraid. It was good to enable CONFIG_AUFS_DEBUG.
>
> I don't know why smack people designed such behaviour. That is
> inetresting. At the same time, I think it ugly. Why didn't they create a
> hook at the very end of lookup, I wonder.
>
> Anyway here is a quick hack for you. Won't you test it?
>
>
> J. R. Okajima
>

Thank you for the quick reply.
I have tested the patch. 'ls' behavior is the same, it hangs. There
are no error messages coming now may be it is doing something inside
au_h_path_getattr() with holding the lock (When printed locked is -1).

--Arun

[s***@users.sourceforge.net]

Arun Chandran:
> Thank you for the quick reply.
> I have tested the patch. 'ls' behavior is the same, it hangs. There
> are no error messages coming now may be it is doing something inside
> au_h_path_getattr() with holding the lock (When printed locked is -1).

Ah, it was necessary for au_h_path_getattr() to treat similarly.
Please revert the previous patch and try this one.


J. R. Okajima

[Arun Chandran]

On Sat, Jan 7, 2017 at 6:51 PM, <***@users.sourceforge.net> wrote:
> Arun Chandran:
>> Thank you for the quick reply.
>> I have tested the patch. 'ls' behavior is the same, it hangs. There
>> are no error messages coming now may be it is doing something inside
>> au_h_path_getattr() with holding the lock (When printed locked is -1).
>
> Ah, it was necessary for au_h_path_getattr() to treat similarly.
> Please revert the previous patch and try this one.
>
>
> J. R. Okajima
>

Now it seems fine.

# cd /mnt/mnt/
# ls
a1.txt a2.txt a3.txt b1.txt b2.txt b3.txt c1.txt c2.txt c3.txt
#
# for i in `ls`
> do
> chsmack $i
> done
a1.txt access="k1"
a2.txt access="k1"
a3.txt access="k1"
b1.txt access="k1"
b2.txt access="k1"
b3.txt access="k1"
c1.txt access="k1"
c2.txt access="k1"
c3.txt access="k1"
#

Are you going to fix this in your tree? Please feel free to ask my
help in testing it, I will happily do it.

--Arun

[s***@users.sourceforge.net]

Arun Chandran:
> Are you going to fix this in your tree? Please feel free to ask my
> help in testing it, I will happily do it.

Thanx for testing.
As you might know, aufs3.19 is not maintained now. So the tree won't be
updated anymore. But similar issue will happen in aufs4.1 and later. I
mean the supported branches. After a few more considerations and
refining the patch, I MAY merge the patch into aufs4.1 and later. But
aufs3.19 won't be updated.


J. R. Okajima

[Arun Chandran]

On Sat, Jan 7, 2017 at 7:51 PM, <***@users.sourceforge.net> wrote:
>
> Arun Chandran:
>> Are you going to fix this in your tree? Please feel free to ask my
>> help in testing it, I will happily do it.
>
> Thanx for testing.
> As you might know, aufs3.19 is not maintained now. So the tree won't be
> updated anymore. But similar issue will happen in aufs4.1 and later. I
> mean the supported branches. After a few more considerations and
> refining the patch, I MAY merge the patch into aufs4.1 and later. But
> aufs3.19 won't be updated.
>
>
> J. R. Okajima
Hi,

Now I moved my testing to 4.1.17(Used origin/aufs4.1.13+ from
aufs4-standalone + b.patch applied).

As a root user mounted the layers:
# mount -t aufs -o
br=./layer2=rw:./layer1=ro:./layer0=ro:,noplink,smackfsdef=k1,smackfsroot=k1
-o udba=reval none ./rootfs_mnt/


As a normal user
# id
uid=1001(test) gid=1001(test) groups=1001(test)
# find layer*
layer0
layer0/etc
layer0/etc/0_3.txt
layer0/etc/0_2.txt
layer0/etc/0_1.txt
layer1
layer1/etc
layer1/etc/1_3.txt
layer1/etc/1_1.txt
layer1/etc/1_2.txt
layer2
layer2/.wh..wh.aufs
layer2/root
layer2/root/2_1.txt
layer2/root/2_3.txt
layer2/root/2_2.txt
layer2/.wh..wh.orph
find: layer2/.wh..wh.orph: Permission denied
#
# chsmack layer2/.wh..wh.orph
layer2/.wh..wh.orph access="_"
# chsmack layer2/.wh..wh.aufs
layer2/.wh..wh.aufs access="_"
#
# cd rootfs_mnt/etc/
#
# rm *
rm: can't remove '0_1.txt': Permission denied
rm: can't remove '0_2.txt': Permission denied
rm: can't remove '0_3.txt': Permission denied
rm: can't remove '1_1.txt': Permission denied
rm: can't remove '1_2.txt': Permission denied
rm: can't remove '1_3.txt': Permission denied
#
# cat /proc/self/attr/current
k1
#------------------> In another root terminal executed '# chsmack -a
k1 layer2/.wh..wh.aufs'
# rm *
---------------------> rm succeeded

Below are the audit messages from smack during the failure
###############
# dmesg -c
[ 857.268080] audit: type=1400 audit(1484226775.112:3): lsm=SMACK
fn=smack_inode_link action=denied subject="k1" object="_" requested=w
pid=180 comm="rm" name=".wh..wh.aufs" dev="hda1" ino=806
[ 857.304931] audit: type=1400 audit(1484226775.150:4): lsm=SMACK
fn=smack_inode_link action=denied subject="k1" object="_" requested=w
pid=180 comm="rm" name=".wh..wh.aufs" dev="hda1" ino=806
[ 857.318879] audit: type=1400 audit(1484226775.164:5): lsm=SMACK
fn=smack_inode_link action=denied subject="k1" object="_" requested=w
pid=180 comm="rm" name=".wh..wh.aufs" dev="hda1" ino=806
[ 857.327849] audit: type=1400 audit(1484226775.173:6): lsm=SMACK
fn=smack_inode_link action=denied subject="k1" object="_" requested=w
pid=180 comm="rm" name=".wh..wh.aufs" dev="hda1" ino=806
################


Does that mean 'smackfsdef=k1' supplied while mouting don't have any
influence on the labels selected ("_") for
layer2/.wh..wh.aufs and layer2/.wh..wh.orph ?


--Arun

[s***@users.sourceforge.net]

Arun Chandran:
> Now I moved my testing to 4.1.17(Used origin/aufs4.1.13+ from
> aufs4-standalone + b.patch applied).
>
> As a root user mounted the layers:
> # mount -t aufs -o
> br=./layer2=rw:./layer1=ro:./layer0=ro:,noplink,smackfsdef=k1,smackfsroot=k1
> -o udba=reval none ./rootfs_mnt/
:::
> Does that mean 'smackfsdef=k1' supplied while mouting don't have any
> influence on the labels selected ("_") for
> layer2/.wh..wh.aufs and layer2/.wh..wh.orph ?

Hmm, it seems that I have to learn about smack. Honestly speaking I do
know nothing about that.
Anyway what is smackfsdef? It that a mount option for smackfs? If so, it
has no effect. Or is it a generic mount option for any filesystem? If
so, it should have an effect on aufs as well.
What will happen if you specify smackfsdef for layer[012]?


J. R. Okajima

[Arun Chandran]

On Sat, Jan 14, 2017 at 11:20 AM, <***@users.sourceforge.net> wrote:
>
> Arun Chandran:
>> Now I moved my testing to 4.1.17(Used origin/aufs4.1.13+ from
>> aufs4-standalone + b.patch applied).
>>
>> As a root user mounted the layers:
>> # mount -t aufs -o
>> br=./layer2=rw:./layer1=ro:./layer0=ro:,noplink,smackfsdef=k1,smackfsroot=k1
>> -o udba=reval none ./rootfs_mnt/
> :::
>> Does that mean 'smackfsdef=k1' supplied while mouting don't have any
>> influence on the labels selected ("_") for
>> layer2/.wh..wh.aufs and layer2/.wh..wh.orph ?
>
> Hmm, it seems that I have to learn about smack. Honestly speaking I do
> know nothing about that.
> Anyway what is smackfsdef? It that a mount option for smackfs? If so, it
> has no effect. Or is it a generic mount option for any filesystem? If
> so, it should have an effect on aufs as well.
> What will happen if you specify smackfsdef for layer[012]?
>
>
> J. R. Okajima

Me too is not an expert on smack; just started using it now.
------------------
>From the doc:
------------------
Smack supports some mount options:

smackfsdef=label: specifies the label to give files that lack
the Smack label extended attribute.

smackfsroot=label: specifies the label to assign the root of the
file system if it lacks the Smack extended attribute.

smackfshat=label: specifies a label that must have read access to
all labels set on the filesystem. Not yet enforced.

smackfsfloor=label: specifies a label to which all labels set on the
filesystem must have read access. Not yet enforced.


--------------------------------------------------------------------------------------------
"k1" is the smack label given to all the branches and files in my setup
--------------------------------------------------------------------------------------------
# for i in `find layer*`; do chsmack $i; done
layer0 access="k1"
layer0/etc access="k1"
layer0/etc/0_3.txt access="k1"
layer0/etc/0_2.txt access="k1"
layer0/etc/0_1.txt access="k1"
layer1 access="k1"
layer1/etc access="k1"
layer1/etc/1_3.txt access="k1"
layer1/etc/1_1.txt access="k1"
layer1/etc/1_2.txt access="k1"
layer2 access="k1"
layer2/root access="k1"
layer2/root/2_1.txt access="k1"
layer2/root/2_3.txt access="k1"
layer2/root/2_2.txt access="k1"

I am running as a user "test" labeled as 'k1'

# id
uid=1001(test) gid=1001(test) groups=1001(test)
# cat /proc/self/attr/current
k1#

This means the subject (user test) has all the access (rwx) to objects
labelled as "k1"

-------------------------------------------------------------
Now from a root terminal I mount the branches
--------------------------------------------------------------

# mount -t aufs -o br=./layer2=rw:./layer1=ro:./layer0=ro -o
udba=reval -o noplink,smackfsdef=k1,smackfsroot=k1 none ./rootfs_mnt
#
# chsmack rootfs_mnt/
rootfs_mnt/ access="k1"
#
# id
uid=0(root) gid=0(root) groups=0(root),10(wheel)
#
# pwd
/home/test

----------------------------------------
After the mount the labels are
----------------------------------------

# for i in `find layer*`; do chsmack $i; done
find: layer2/.wh..wh.orph: Permission denied
layer0 access="k1"
layer0/etc access="k1"
layer0/etc/0_3.txt access="k1"
layer0/etc/0_2.txt access="k1"
layer0/etc/0_1.txt access="k1"
layer1 access="k1"
layer1/etc access="k1"
layer1/etc/1_3.txt access="k1"
layer1/etc/1_1.txt access="k1"
layer1/etc/1_2.txt access="k1"
layer2 access="k1"
layer2/.wh..wh.aufs access="_"
layer2/root access="k1"
layer2/root/2_1.txt access="k1"
layer2/root/2_3.txt access="k1"
layer2/root/2_2.txt access="k1"
layer2/.wh..wh.orph access="_"
#

The labels given to layer2/.wh..wh.orph and layer2/.wh..wh.orph is "_"
because it is formed by mount (running as root)
As the objects gets the label of the subject (root now) those files
got the labels "_". I thought they should be getting the labels from
"smackfsdef" and it seems I am wrong.

-----------------------------------------------------------------
User test unable to delete files from the mount dir
------------------------------------------------------------------

# cd rootfs_mnt/
#
# rm etc/*
rm: can't remove 'etc/0_1.txt': Permission denied
rm: can't remove 'etc/0_2.txt': Permission denied
rm: can't remove 'etc/0_3.txt': Permission denied
rm: can't remove 'etc/1_1.txt': Permission denied
rm: can't remove 'etc/1_2.txt': Permission denied
rm: can't remove 'etc/1_3.txt': Permission denied
#
#
# dmesg
[ 3429.784711] audit: type=1400 audit(1484377632.643:2): lsm=SMACK
fn=smack_inode_link action=denied subject="k1" object="_" requested=w
pid=409 comm="rm" name=".wh..wh.aufs" dev="hda1" ino=806
[ 3429.813012] audit: type=1400 audit(1484377632.671:3): lsm=SMACK
fn=smack_inode_link action=denied subject="k1" object="_" requested=w
pid=409 comm="rm" name=".wh..wh.aufs" dev="hda1" ino=806
[ 3429.817525] audit: type=1400 audit(1484377632.676:4): lsm=SMACK
fn=smack_inode_link action=denied subject="k1" object="_" requested=w
pid=409 comm="rm" name=".wh..wh.aufs" dev="hda1" ino=806
[ 3429.820807] audit: type=1400 audit(1484377632.678:5): lsm=SMACK
fn=smack_inode_link action=denied subject="k1" object="_" requested=w
pid=409 comm="rm" name=".wh..wh.aufs" dev="hda1" ino=806
[ 3429.830212] audit: type=1400 audit(1484377632.684:6): lsm=SMACK
fn=smack_inode_link action=denied subject="k1" object="_" requested=w
pid=409 comm="rm" name=".wh..wh.aufs" dev="hda1" ino=806
[ 3429.842098] audit: type=1400 audit(1484377632.700:7): lsm=SMACK
fn=smack_inode_link action=denied subject="k1" object="_" requested=w
pid=409 comm="rm" name=".wh..wh.aufs" dev="hda1" ino=806


As of now I can workaround this problem by manually labeling
".wh..wh.aufs" to "k1".

--Arun

[s***@users.sourceforge.net]

Arun Chandran:
> # mount -t aufs -o br=./layer2=rw:./layer1=ro:./layer0=ro -o
> udba=reval -o noplink,smackfsdef=k1,smackfsroot=k1 none ./rootfs_mnt

I'd suggest you to specify and test smack options on layerN instead of
rootfs_mnt.
Aufs is a virtual filesystem and it doesn't have a backend block
device. Which means it doesn't store smack attributes to the disk. All
attributes are handled and stored by the branch fs (layner[0-2] in your
case).


> # chsmack rootfs_mnt/
> rootfs_mnt/ access="k1"

So this k1 value is taken from the topmost branch (./layer2 in your
case).

Here is what I am thinking/guessing,
- smack mount options try setting the default attribute to rootfs_mnt,
which will be applied to all files under rootfs_mnt.
- rootfs_mnt is aufs, I mean "virtual".
- all attributes in aufs are inherited from the branch fs.
- even if you and smack set attribute to virtual aufs, it will be
overwritten by the real one on the branch fs.
Again I'd suggest you to set smack options to your branches instead of
aufs.

Let's consider a further case.
- /u = /rw + /ro
- specify smackfsdef=privA to /rw
- specify smackfsdef=privB to /u
and then
case1: access /rw/fileC as smack privA
case2: access /rw/fileC as smack privB
case3: access /u/fileC as smack privA
case4: access /u/fileC as smack privB

case1 shoule be allowed obviously.
case2 will be rejected, since /rw accepts privA only.

case3, to be or not to be? That is the question.
If we follow a principle that /u accepts privB only, then case3 should
be rejected. But internally this behaviour concludes an access to
/rw/fileC. As long as /rw accepts privA, case3 should be allowed based
upon /rw's point of view.
case4 is just a counter of case3.

As a very fundamental design, aufs tries behaving as the branch fs would
do (as possible). All accesses go to the branches, and all data is
retrived from the branches. So my (current) conclusion is
- setting smackfsdef to aufs has no meaning.
- the effective smack attribute is the one set to the branch fs.

How do you think?


J. R. Okajima

[Arun Chandran]

On Sun, Jan 15, 2017 at 1:04 AM, Casey Schaufler <***@schaufler-ca.com> wrote:
> On 1/14/2017 6:15 AM, ***@users.sourceforge.net wrote:
>> Arun Chandran:
>
> Thank you for using Smack!
>
>>> # mount -t aufs -o br=./layer2=rw:./layer1=ro:./layer0=ro -o
>>> udba=reval -o noplink,smackfsdef=k1,smackfsroot=k1 none ./rootfs_mnt
>
> The smackfsdef option specifies the Smack label to use if
> there is no security.SMACK64 xattr available for an object
> on the filesystem. Note that the active Smack label on an
> object is stored in the inode, so if an object is created
> on any filesystem it will get the Smack label of the creating
> process*. If the filesystem does not support xattrs the Smack
> label will be lost on reboot.
>
>> I'd suggest you to specify and test smack options on layerN instead of
>> rootfs_mnt.
>> Aufs is a virtual filesystem and it doesn't have a backend block
>> device. Which means it doesn't store smack attributes to the disk. All
>> attributes are handled and stored by the branch fs (layner[0-2] in your
>> case).
>>
>>
>>> # chsmack rootfs_mnt/
>>> rootfs_mnt/ access="k1"
>> So this k1 value is taken from the topmost branch (./layer2 in your
>> case).
>>
>> Here is what I am thinking/guessing,
>> - smack mount options try setting the default attribute to rootfs_mnt,
>> which will be applied to all files under rootfs_mnt.
>> - rootfs_mnt is aufs, I mean "virtual".
>> - all attributes in aufs are inherited from the branch fs.
>> - even if you and smack set attribute to virtual aufs, it will be
>> overwritten by the real one on the branch fs.
>> Again I'd suggest you to set smack options to your branches instead of
>> aufs.
>
> I'm not well versed in aufs, but if it's like cryptfs or
> overlayfs some of the details of how xattrs are maintained
> may be mode complicated than they might seem. With that
> caution in mind, your description above ought to hold.
>
>> Let's consider a further case.
>> - /u = /rw + /ro
>> - specify smackfsdef=privA to /rw
>> - specify smackfsdef=privB to /u
>> and then
>> case1: access /rw/fileC as smack privA
>> case2: access /rw/fileC as smack privB
>> case3: access /u/fileC as smack privA
>> case4: access /u/fileC as smack privB
>>
>> case1 shoule be allowed obviously.
>> case2 will be rejected, since /rw accepts privA only.
>>
>> case3, to be or not to be? That is the question.
>> If we follow a principle that /u accepts privB only, then case3 should
>> be rejected. But internally this behaviour concludes an access to
>> /rw/fileC. As long as /rw accepts privA, case3 should be allowed based
>> upon /rw's point of view.
>> case4 is just a counter of case3.
>>
>> As a very fundamental design, aufs tries behaving as the branch fs would
>> do (as possible). All accesses go to the branches, and all data is
>> retrived from the branches. So my (current) conclusion is
>> - setting smackfsdef to aufs has no meaning.
>> - the effective smack attribute is the one set to the branch fs.
>>
>> How do you think?
>
> Assuming all the mechanics are in order that is
> what I would expect.
>

Thanks for the nice explanation.
so 'smackfsdef' for files that don't have support to store label. In
my case I have all the branches(layers) and files labelled as 'k1' and
the underlying filesystem(ext4) supports the extended attributes.

Please see the files and their attributes below.
----------------------------------------------
# for i in `find layer*`; do chsmack $i; done
layer0 access="k1"
layer0/0.txt access="k1"
layer1 access="k1"
layer1/1.txt access="k1"
#
# chsmack rootfs_mnt/
rootfs_mnt/ access="k1"
# pwd
/home/test
# cat /proc/self/attr/current
k1
# chsmack .
. access="k1"
#
# id
uid=1001(test) gid=1001(test) groups=1001(test)
#
----------------------------------------------------

All I want is to do aufs mount layer0 and layer1 and perform normal
filesytem operations there.

Now I mount the layers. As I can't mount as user test, I execute the
mount in a root terminal
# mount -t aufs -o br=./layer1=rw:./layer0=ro -o udba=reval none -o
smackfsroot=k1 ./rootfs_mnt
# id
uid=0(root) gid=0(root) groups=0(root),10(wheel)

When I try to delete the files as a user test
# cd rootfs_mnt/
# ls
0.txt 1.txt
# rm *
rm: can't remove '0.txt': Permission denied
#
# ls
0.txt

This happens because aufs handles removal of files through newly
created file "layer1/.wh..wh.aufs"(I am guessing this from the below
printk), as this file got created during the mount operation it is
labeled as "_"

[ 3365.600800] audit: type=1400 audit(1484470779.482:23): lsm=SMACK
fn=smack_inode_link action=denied subject="k1" object="_" requested=w
pid=385 comm="rm" name=".wh..wh.aufs" dev="hda1" ino=796

In short words, what I am looking is a mount option like this
"aufs_meta_files_def_smack_label=k1" :).
Could you please give me pointer to how to implement such an option?

--Arun

[s***@users.sourceforge.net]

Arun Chandran:
> This happens because aufs handles removal of files through newly
> created file "layer1/.wh..wh.aufs"(I am guessing this from the below
> printk), as this file got created during the mount operation it is
> labeled as "_"

Still I don't get the point.
Would you try these steps? I want to know what will happen on layer1.

cd ./layer1
> .wh..wh.aufs (if it doesn't exist)
ln .wh..wh.aufs .wh.0.txt


> In short words, what I am looking is a mount option like this
> "aufs_meta_files_def_smack_label=k1" :).
> Could you please give me pointer to how to implement such an option?

What will this option does?


J. R. Okajima

[Arun Chandran]

On Mon, Jan 16, 2017 at 12:08 AM, <***@users.sourceforge.net> wrote:
>
> Arun Chandran:
>> This happens because aufs handles removal of files through newly
>> created file "layer1/.wh..wh.aufs"(I am guessing this from the below
>> printk), as this file got created during the mount operation it is
>> labeled as "_"
>
> Still I don't get the point.
> Would you try these steps? I want to know what will happen on layer1.
>
> cd ./layer1
>> .wh..wh.aufs (if it doesn't exist)
> ln .wh..wh.aufs .wh.0.txt
>
>
---------------------------------------------
Files in the layers before mounting
----------------------------------------------
# for i in `find layer* `; do chsmack $i; done
layer0 access="k1"
layer0/0.txt access="k1"
layer1 access="k1"
layer1/1.txt access="k1"

------------------------------------------------------------
Given below are the files I can see in the layers after doing aufs
mount of layer0 and layer1 to rootfs_mnt.
Please note that I am running as a normal user 'test'.
------------------------------------------------------------
# find layer*
layer0
layer0/0.txt
layer1
layer1/1.txt
layer1/.wh..wh.aufs
layer1/.wh..wh.plnk
find: layer1/.wh..wh.plnk: Permission denied
layer1/.wh..wh.orph
find: layer1/.wh..wh.orph: Permission denied
-------------------------------------------------------
Now printing their individual smack labels
--------------------------------------------------------
# for i in `find layer* `; do chsmack $i; done
find: layer1/.wh..wh.plnk: Permission denied
find: layer1/.wh..wh.orph: Permission denied
layer0 access="k1"
layer0/0.txt access="k1"
layer1 access="k1"
layer1/1.txt access="k1"
layer1/.wh..wh.aufs access="_" --------------> These meta data files
are labelled "_"
layer1/.wh..wh.plnk access="_"
layer1/.wh..wh.orph access="_"
#

mounting layer0 and layer1 to rootfs_mnt is done in a root terminal
with the command
'mount -t aufs -o br=./layer1=rw:./layer0=ro -o udba=reval -o
smackfsroot=k1 none ./rootfs_mnt'

Am I clear now?

>> In short words, what I am looking is a mount option like this
>> "aufs_meta_files_def_smack_label=k1" :).
>> Could you please give me pointer to how to implement such an option?
>
> What will this option does?

This option will chose the supplied smack label (k1) for the newly
created aufs meta data files.
In my case you can see the aufs meta data files (.wh..*.*) are labelled as "_".

[This is happening because new objects get the label of subject. These
files are created while mounting as a root user. The smack label for
root user is "_". ]

--Arun

[s***@users.sourceforge.net]

Arun Chandran:
> ---------------------------------------------
> Files in the layers before mounting
> ----------------------------------------------
> # for i in `find layer* `; do chsmack $i; done
> layer0 access="k1"
> layer0/0.txt access="k1"
> layer1 access="k1"
> layer1/1.txt access="k1"

At this point, won't you try these steps?
If a normal user cannot complete them, try as a superuser too.
Of course I am assuming the smack options are set to layer1.

cd ./layer1
> .wh..wh.aufs
ln .wh..wh.aufs .wh.0.txt


> layer1/.wh..wh.aufs access="_" --------------> These meta data files
> are labelled "_"
> layer1/.wh..wh.plnk access="_"
> layer1/.wh..wh.orph access="_"
:::
> Am I clear now?

No.
- Is the root cause of your problem that .wh..wh.aufs has access="_"?
- If so, why does it have access="_"? A simple creat(2) doesn't set the
attribute you want?
- If creat(2) is not good enough, then what is the correct way?

In order to give the light to these questions, I am asking the test of
creat(2) and link(2) manually. How can you get the result you want
manually?


> This option will chose the supplied smack label (k1) for the newly
> created aufs meta data files.
> In my case you can see the aufs meta data files (.wh..*.*) are labelled as "_".
>
> [This is happening because new objects get the label of subject. These
> files are created while mounting as a root user. The smack label for
> root user is "_". ]

I am afraid such option violates the security policy of branch fs's or
smacks's. If it is acceptable, then why don't you set other label to
.wh..wh.aufs manually after mount(2)?

Is this a natural-born-problem from a mixed use-case of smack and how
you mount aufs?
- the smack label for root user is "_".
- aufs creates .wh..wh.aufs at mounting.
- you mount aufs as root.

Then .wh..wh.aufs has "_" obviously. And there may exist another way you
can take such as,
- mount aufs by a user other than root.
+ that user has to have a capability to isseu mount(2) and other
necessary syscalls.
+ the smack label for that user should not be "_".

If necesssary, I will refine the capability in aufs. It will be much
better than violating the security policy.


J. R. Okajima

[Arun Chandran]

On Mon, Jan 16, 2017 at 1:19 AM, <***@users.sourceforge.net> wrote:
>
> Arun Chandran:
>> ---------------------------------------------
>> Files in the layers before mounting
>> ----------------------------------------------
>> # for i in `find layer* `; do chsmack $i; done
>> layer0 access="k1"
>> layer0/0.txt access="k1"
>> layer1 access="k1"
>> layer1/1.txt access="k1"
>
> At this point, won't you try these steps?
> If a normal user cannot complete them, try as a superuser too.
> Of course I am assuming the smack options are set to layer1.
>
> cd ./layer1
>> .wh..wh.aufs
> ln .wh..wh.aufs .wh.0.txt
>

Please find the details below

# id
uid=1001(test) gid=1001(test) groups=1001(test)
# cat /proc/self/status | grep -i cap
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003fffffffff
# cat /proc/self/attr/current
k1#
# mkdir layer0 layer1 rootfs_mnt
# echo 0 >layer0/0.txt
# echo 1 >layer1/1.txt
#
# for i in `find layer*`
> do
> chsmack $i
> done
layer0 access="k1"
layer0/0.txt access="k1"
layer1 access="k1"
layer1/1.txt access="k1"
#
# cd layer1/
# >.wh..wh.aufs
# ln .wh..wh.aufs .wh.0.txt
#
# chsmack .wh..wh.aufs
.wh..wh.aufs access="k1"
# chsmack .wh.0.txt
.wh.0.txt access="k1"


# chsmack rootfs_mnt/
rootfs_mnt/ access="k1"


Now I mount the branches to rootfs_mnt in a root terminal as below
mount -t aufs -o br=./layer1=rw:./layer0=ro -o udba=reval -o
smackfsroot=k1 none ./rootfs_mnt


As user test:
-----------------

# for i in `find layer*`; do chsmack $i; done
find: layer1/.wh..wh.plnk: Permission denied
find: layer1/.wh..wh.orph: Permission denied
layer0 access="k1"
layer0/0.txt access="k1"
layer1 access="k1"
layer1/1.txt access="k1"
layer1/.wh.0.txt access="k1"
layer1/.wh..wh.aufs access="k1"
layer1/.wh..wh.plnk access="_"
layer1/.wh..wh.orph access="_"
#
# cd rootfs_mnt/
#
# ls
1.txt
#
# rm 1.txt
#

>
>> layer1/.wh..wh.aufs access="_" --------------> These meta data files
>> are labelled "_"
>> layer1/.wh..wh.plnk access="_"
>> layer1/.wh..wh.orph access="_"
> :::
>> Am I clear now?
>
> No.
Sorry for not making it clear enough.

> - Is the root cause of your problem that .wh..wh.aufs has access="_"?
> - If so, why does it have access="_"?

As the .wh..* files are created during mount they inherit the smack
label of root. so they get access="_".

> A simple creat(2) doesn't set the
> attribute you want?
> - If creat(2) is not good enough, then what is the correct way?

Yes I can do a chown + chmod to .wh* files to make it accessible to
user 'test'. Also I need to change their labels to "k1" because
1) User test has label "k1"
2) User test can't change the labels on its own as it lacks CAP_MAC_ADMIN
3) SMACK does not allow subjects (process with UID test) with label
"k1" to do write operation on objects with label "_"

> In order to give the light to these questions, I am asking the test of
> creat(2) and link(2) manually. How can you get the result you want
> manually?

>From the root terminal I can change labels of all the .wh* files and
can also change their ownership to normal user.
This can be performed after the mount operation and it will allow me
to get the desired result.

Do you see any security risk in doing so?
>
>
>> This option will chose the supplied smack label (k1) for the newly
>> created aufs meta data files.
>> In my case you can see the aufs meta data files (.wh..*.*) are labelled as "_".
>>
>> [This is happening because new objects get the label of subject. These
>> files are created while mounting as a root user. The smack label for
>> root user is "_". ]
>
> I am afraid such option violates the security policy of branch fs's or
> smacks's. If it is acceptable, then why don't you set other label to
> .wh..wh.aufs manually after mount(2)?

Yes I am using this workaround now. Manually labeling the .wh* files

>
> Is this a natural-born-problem from a mixed use-case of smack and how
> you mount aufs?

Is their any way of differing the formation of .wh.* files till
somebody really starts doing some
file operations? In that way those files will always get the label of
the guy who is doing the operation.


The outline of our plan is given below

|container0 | |containerrN |
|Docker App0 | |Docker AppN |
|Everything inside| - - |Everything inside|
|labelled as k0 | |labelled as kN |
|-----------------------| |----------------------|

root user will label the layers and do aufs mount of the layers before
starting the container. The processes running inside the container
will also have a unique label such as k0,k1,... kN and they should be
able to do any kind of operations in their respective aufs mounted
directories.

--Arun

[s***@users.sourceforge.net]

Arun Chandran:
> # id
> uid=1001(test) gid=1001(test) groups=1001(test)
:::
> # cd layer1/
> # >.wh..wh.aufs
> # ln .wh..wh.aufs .wh.0.txt

Ok, succeeded with a normal user.
How about as a superuser?

cd layer1
sudo touch .wh..wh.aufs
ln .wh..wh.aufs .wh.0.txt

Is .wh..wh.aufs created with access="_"?
Is .wh.0.txt failed to be linked?


> From the root terminal I can change labels of all the .wh* files and
> can also change their ownership to normal user.
> This can be performed after the mount operation and it will allow me
> to get the desired result.
>
> Do you see any security risk in doing so?

It seems breaking what smack is trying to protect by access="_". I don't
know what it is. But as long as it is acceptable for you and you can get
the desired result, it might be a good way to go.


> Is their any way of differing the formation of .wh.* files till
> somebody really starts doing some
> file operations? In that way those files will always get the label of
> the guy who is doing the operation.

It is up to how the smack label is set.
Note that aufs doesn't care about smack settings, and just follows the
behaviour of its branch fs's and smack's. As long as
- smack sets access="_" to the files which a superuser created.
- you mount aufs as a superuser.
then, the symptom looks a correct result.
If you just want to set access= other than "_", then you can do it by
either resetting after mount or changing the mount-user other than root
(based on the capability).


> root user will label the layers and do aufs mount of the layers before
> starting the container. The processes running inside the container
> will also have a unique label such as k0,k1,... kN and they should be
> able to do any kind of operations in their respective aufs mounted
> directories.

So all the files which are set the label are unique to the container
respectively, right? There will be no files shared between cocntainers
at all? Or a single file can have multiple smack labels?


J. R. Okajima

[Arun Chandran]

On Mon, Jan 16, 2017 at 12:29 PM, <***@users.sourceforge.net> wrote:
>
> Arun Chandran:
>> # id
>> uid=1001(test) gid=1001(test) groups=1001(test)
> :::
>> # cd layer1/
>> # >.wh..wh.aufs
>> # ln .wh..wh.aufs .wh.0.txt
>
> Ok, succeeded with a normal user.
> How about as a superuser?
>
> cd layer1
> sudo touch .wh..wh.aufs
> ln .wh..wh.aufs .wh.0.txt
>
> Is .wh..wh.aufs created with access="_"?
> Is .wh.0.txt failed to be linked?
>
No, It succeeded and created with label "k1", please see below
# mkdir layer0 layer1 rootfs_mnt
# echo 0 > layer0/0.txt
# echo 1 > layer1/1.txt
#
# cd layer1/
# sudo touch .wh..wh.aufs
# ln .wh..wh.aufs .wh.0.txt
#
./.wh.0.txt access="k1"
./.wh..wh.aufs access="k1"
./1.txt access="k1"

>
>> From the root terminal I can change labels of all the .wh* files and
>> can also change their ownership to normal user.
>> This can be performed after the mount operation and it will allow me
>> to get the desired result.
>>
>> Do you see any security risk in doing so?
>
> It seems breaking what smack is trying to protect by access="_". I don't
> know what it is. But as long as it is acceptable for you and you can get
> the desired result, it might be a good way to go.
>
>
>> Is their any way of differing the formation of .wh.* files till
>> somebody really starts doing some
>> file operations? In that way those files will always get the label of
>> the guy who is doing the operation.
>
> It is up to how the smack label is set.
> Note that aufs doesn't care about smack settings, and just follows the
> behaviour of its branch fs's and smack's. As long as
> - smack sets access="_" to the files which a superuser created.
> - you mount aufs as a superuser.
> then, the symptom looks a correct result.
> If you just want to set access= other than "_", then you can do it by
> either resetting after mount or changing the mount-user other than root
> (based on the capability).

OK. I will have to manually change the labels for aufs meta data files (.wh.*).
Only the root user will have the necessary capabilities, all other
users will be having limited capabilities.

>
>> root user will label the layers and do aufs mount of the layers before
>> starting the container. The processes running inside the container
>> will also have a unique label such as k0,k1,... kN and they should be
>> able to do any kind of operations in their respective aufs mounted
>> directories.
>
> So all the files which are set the label are unique to the container
> respectively, right? There will be no files shared between cocntainers
> at all? Or a single file can have multiple smack labels?

Yes. There won't be any sharing of files between containers; all the files in
a container will have unique label which is different from the other containers.

A single file can have only one label.

--Arun

[s***@users.sourceforge.net]

Arun Chandran:
> No, It succeeded and created with label "k1", please see below

Ok, then let's make sure again.
- you wrote that the smack label for root user is "_".
- "sudo mount -t aufs ..." created the file with access="_".
- "sudo touch ..." created with access="k1".

Why didn't "touch" and simple creat(2) set access="_"?
I am confusing now and don't know what is the correct behaviour.
Either I don't know there is something to do left in aufs or smack.


J. R. Okajima

[Arun Chandran]

On Mon, Jan 16, 2017 at 6:39 PM, <***@users.sourceforge.net> wrote:
>
> Arun Chandran:
>> No, It succeeded and created with label "k1", please see below
>
> Ok, then let's make sure again.
> - you wrote that the smack label for root user is "_".
Yes. That is correct.
# id
uid=0(root) gid=0(root) groups=0(root),10(wheel)
# cat /proc/self/attr/current
_

> - "sudo mount -t aufs ..." created the file with access="_".
No with 'sudo mount ..' the .wh.* files are created with label of the
user test not with the label of root.
[This is because objects gets label of the process; label of user test
is "k1"; sudo is not changing label]

sudo mount and labels:
------------------------------
# mkdir layer0 layer1
# echo 0 > layer0/0.txt
# echo 1 > layer1/1.txt
# sudo mount -t aufs -o br=./layer1=rw:./layer0=ro -o udba=reval -o
smackfsroot=k1 none ./rootfs_mnt
Password:
# for i in `find layer*`; do chsmack $i; done
find: 'layer1/.wh..wh.plnk': Permission denied
find: 'layer1/.wh..wh.orph': Permission denied
layer0 access="k1"
layer0/0.txt access="k1"
layer1 access="k1"
layer1/.wh..wh.plnk access="k1"
layer1/.wh..wh.aufs access="k1"
layer1/1.txt access="k1"
layer1/.wh..wh.orph access="k1"
#
# id
uid=1001(test) gid=1001(test) groups=1001(test)
#

root mount and labels:
------------------------------
All the steps same as above except mounted rootfs_mnt from root terminal.

# for i in `find layer*`; do chsmack $i; done
layer0 access="k1"
layer0/0.txt access="k1"
layer1 access="k1"
layer1/.wh..wh.plnk access="_"
layer1/.wh..wh.aufs access="_"
layer1/1.txt access="k1"
layer1/.wh..wh.orph access="_"
# id
uid=0(root) gid=0(root) groups=0(root),10(wheel)
#
# chsmack rootfs_mnt/
rootfs_mnt/ access="k1"
#

--Arun

[s***@users.sourceforge.net]

Arun Chandran:
> No with 'sudo mount ..' the .wh.* files are created with label of the
> user test not with the label of root.
> [This is because objects gets label of the process; label of user test
> is "k1"; sudo is not changing label]

I see.
It may be a very basic building block of security label NOT to use the
effective uid.

Back in our simple tests,

cd layer1/
> .wh..wh.aufs
ln .wh..wh.aufs .wh.0.txt

- by a normal user, .wh..wh.aufs will have access="k1".
- sudo by a normal user, it will be access="k1" too.
- by a plain superuser, it will be access="_".
right?

And "sudo mount" sets access="k1" to .wh..wh.aufs.
Good. It must be the way to go, isn't it?


J. R. Okajima

[Arun Chandran]

On Mon, Jan 16, 2017 at 7:59 PM, <***@users.sourceforge.net> wrote:
>
> Arun Chandran:
>> No with 'sudo mount ..' the .wh.* files are created with label of the
>> user test not with the label of root.
>> [This is because objects gets label of the process; label of user test
>> is "k1"; sudo is not changing label]
>
> I see.
> It may be a very basic building block of security label NOT to use the
> effective uid.
>
> Back in our simple tests,
>
> cd layer1/
>> .wh..wh.aufs
> ln .wh..wh.aufs .wh.0.txt
>
> - by a normal user, .wh..wh.aufs will have access="k1".
> - sudo by a normal user, it will be access="k1" too.
> - by a plain superuser, it will be access="_".
> right?
>
Yes. Correct.

> And "sudo mount" sets access="k1" to .wh..wh.aufs.
> Good. It must be the way to go, isn't it?

"sudo mount .." gives correct labels. I can't use it because the
containers don't get sudo inside ; container might be running with the
lowest possible privileges.

I will be doing(as root).

1) Takes the request to load the docker app
2) Label the all the layers in the app(files) with a smack label "kN"
3) All the layers are aufs mounted
4) Perform a label change of the .wh.* files from "_" to "kN"
5) Change the ownership of the .wh.* files from root to container UID
6) start the container

--Arun

[s***@users.sourceforge.net]

Arun Chandran:
> "sudo mount .." gives correct labels. I can't use it because the
> containers don't get sudo inside ; container might be running with the
> lowest possible privileges.

Our discussion about the smack label is almost done. Thank you.
But I'd suggest you to consider other docker storage drivers, especially
devicemapper because your usecase looks wasting the disk space a lot.
As you might know, aufs executes a file-based-copyup while devicemapper
operates a block-based. In other words, setting a smack label changes
just a small part of an inode, but aufs has to copyup the whole
file. Devmapper changes the inode block only which should be much
smaller than aufs' file-copyup. You are going to have N sets of
files. All file-data are same. Just a label is different. That is why I
suggest re-consider using aufs.
Of course, it depends upon the file sizies and the number of files, or
branch fs' inode block layout. Also I am not sure how you can setup the
containers without sudo.

Anyway "b.path" will be refined and released in next aufs release.


J. R. Okajima