Framework for digital camera in Linux ( )

Framework for digital camera
in Linux
Dongsoo Kim ([email protected])
Heungjun Kim([email protected])
Samsung Electronics
Contents
Introducing mobile camera devices
- SoC camera subsystem
- Mobile camera module devices
Framework for digital camera
- Video4Linux2 and camera device
- New APIs for high-end mobile camera devices
- Dual camera framework for 3G handset
Porting issues in SoC camera subsystem
2
Terms
AP : Application Processor
Camera interface : interface device which is syncing image data
from external camera device and gives some commands to
them
Camera module : external camera device packaged with lens,
sensor and ISP on it
ISP : Image Signal Processor. Also supports lots of additional
functions
V4L2 : Video for Linux 2
3
This document contains some animations.
Slideshow mode is recommended
4
What makes camera “a camera”?
Exposure control
- Aperture (Iris)
- Shutter
- ISO
Strobe support
- Various program
mode : slow sync,
red-eye reduction
and on….
Lens control
- Focus
- Zoom
5
Introducing mobile camera devices
6
History of camera phone
1.3MP
in 2003
First
Camera
Phone
5MP
in2004
3MP
7MP with
in 2004
2005
2MP
in
12MP
in
2009
in 10MP
2000
3.5MP
2006
7
Mobile camera module devices
Old and low-end mobile camera modules
Feature
Exposure
Lens
Strobe
Low end camera
Aperture
Not supported
Shutter
Just change frame rate
ISO
Low sensitivity & High noise level
Focus
Pan focus
Zoom
Digital zoom
Just LED light most of time
8
Mobile camera module devices
Brand new and high-end mobile camera modules
Feature
Exposure
Lens
Strobe
High end camera
Aperture
Supported with lens packaging
Shutter
Supported with lens packaging
ISO
High sensitivity & Low noise level
Focus
Movable focus point (Auto/manual)
Zoom
Optical zoom & Digital zoom
XENON flash is also supported
9
SoC camera subsystem
What is SoC camera subsystem?
ARM SoC solution
Peripherals
Camera interface
Peripheral3 (USB)
ARM
Core
External
Camera
module
Peripheral2 (MMC)
Peripheral1(Display)
10
SoC camera subsystem
Request for data
Image data in expected format
Camera interface
Command
interface
Resolution
handling
Data format
handling
(YUV/RAW/JPEG…)
Buffer
handling
I2C
RAW DATA
Camera module
Exposure
control
Lens
control
White
balance
control
Effect
control
Face
detect
Zoom
control
Strobe
control
.
.
.
11
SoC camera subsystem
Various ways of preview (Live view) in SoC camera subsystems
DATA
ARM
core
Camera
Module
DATA
Camera
Interface
User
Application
M
e
m
o
r
y
COPY!
Frame buffer
OK
Cancel
12
SoC camera subsystem
Various ways of preview (Live view) in SoC camera subsystems
User
Application
DATA
ARM
core
Camera
Module
DATA
Camera
Interface
M
e
m
o
r
y
D
M
A
COPY!
Save
as file
Through
DMA
Frame buffer
OK
Cancel
13
SoC camera subsystem
How to capture still shots in SoC camera subsystem
- Your camera module has a dedicated JPEG encoder on it
Camera module
ARM
core
DATA
JPEG
PROCESS
MEMORY
DATA
RAW
DATA
PROCESS
DATA
Camera
Interface
User
Application
M
e
m
o
r
y
COPY!
Save as
JPEG file
14
SoC camera subsystem
How to capture still shots in SoC camera subsystem
- No JPEG encoder in your camera module
User
Application
DATA
ARM
core
Camera
Module
DATA
Camera
Interface
M
e
m
o
r
y
COPY!
RAW
DATA
Codec S/W
Save as
JPEG file
15
SoC camera subsystem
How to take motion pictures in SoC camera subsystem
COPY!
DATA
ARM
core
Camera
Module
DATA
Camera
Interface
M
e
m
o
r
y
Frame buffer
User
Application
OK
Codec
S/W
Cancel
Save as…
COPY!
16
What we’ve got in Linux to control camera
In Kernel
V4L2 (AKA Video For Linux 2)
- The second version of the Video For Linux API
- Kernel interface for analog radio and video capture and output drivers
In user space
Multimedia middleware
Gstreamer
- Pipeline based multimedia framework written in the C programming
language with the type system based on GObject
OpenMAX
- Cross-platform set of C-language programming interfaces that provides
abstractions for routines especially useful for audio, video, and still images
17
Mobile camera module devices
Is V4L2 enough for
mobile digital
camera modules?
No way
18
Framework for digital camera
19
Video4Linux2 and camera device
Camera subsystem in Linux kernel aspect
V4L2
videobuf
videobuf
-core
V4l2dev
videobufsg-dma
V4l2ioctl
videodev
V4l2-device
V4l2-intdevice
(old)
V4l2subdev
(new)
Camera interface
V4L2 driver
Command
interface
Resolution
handling
Data format handling
(YUV/RAW/JPEG…)
Buffer
handling
Camera module
V4L2 driver
Exposure
control
Lens
control
White
balance
control
Effect
control
Face
detect
Zoom
control
Strobe
control
.
.
.
20
Abstract view of V4L2 working with video capture device
How does it work with V4L2?
Application or
Middleware
Copy memory to userspace
Open device
VIDIOC_QUERYCAP
VIDIOC_S_INPUT
VIDIOC_QBUF
VIDIOC_REQBUF
VIDIOC_QBUF
VIDIOC_DQBUF
VIDIOC_QUERYBUF
VIDIOC_STREAMON
21
New V4L2 APIs for digital camera
Exposure control
22
New V4L2 APIs for exposure control
Need for detailed control in exposure
- We’ve got camera devices with mechanical shutter
- Iris could be handled also
- ISO controlled output is somehow useful
Why not using enhanced exposure control like a
regular digital camera?
23
New V4L2 APIs for exposure control
What we’ve got in mainline V4L2 right now?
#define V4L2_CID_EXPOSURE_AUTO (V4L2_CID_CAMERA_CLASS_BASE+1)
enum v4l2_exposure_auto_type {
V4L2_EXPOSURE_AUTO = 0,
V4L2_EXPOSURE_MANUAL = 1,
V4L2_EXPOSURE_SHUTTER_PRIORITY = 2,
V4L2_EXPOSURE_APERTURE_PRIORITY = 3
};
#define V4L2_CID_EXPOSURE_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+2)
#define V4L2_CID_EXPOSURE_AUTO_PRIORITY (V4L2_CID_CAMERA_CLASS_BASE+3)
No way to control iris
No way to control exposure metering method
24
New V4L2 APIs for exposure control
Additional exposure control
enum
v4l2_exposure_auto_type {
V4L2_EXPOSURE_AUTO = 0,
V4L2_EXPOSURE_MANUAL = 1,
V4L2_EXPOSURE_SHUTTER_PRIORITY = 2,
V4L2_EXPOSURE_APERTURE_PRIORITY = 3,
/* Additional features for digital camera */
V4L2_EXPOSURE_ISO_PRIORITY =4,
+
+
};
#define V4L2_CID_EXPOSURE_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+2)
#define V4L2_CID_EXPOSURE_AUTO_PRIORITY (V4L2_CID_CAMERA_CLASS_BASE+3)
. . .
+#define V4L2_CID_EXPOSURE_LOCK (V4L2_CID_CAMERA_CLASS_BASE+18)
25
New V4L2 APIs for exposure control
+#define V4L2_CID_CAM_APERTURE(V4L2_CID_CAMERA_CLASS_BASE+19)
+#define V4L2_CID_CAM_SHUTTER (V4L2_CID_CAMERA_CLASS_BASE+20)
+#define V4L2_CID_CAM_ISO
(V4L2_CID_CAMERA_CLASS_BASE+21)
Supported aperture stages, shutter speed, and ISO speed could be
different between every different camera module products
Supported values for aperture, shutter and ISO should be made in
V4L2_CTRL_TYPE_MENU
{
.id = V4L2_CID_CAM_APERTURE,
.type = V4L2_CTRL_TYPE_MENU,
.name = "Aperture",
.minimum = 0,
.maximum = 4,
.step = 1,
.default_value = 0,
},
static const char *camera_iris_stages[] = {
/* This module supports 5 Iris stages
* on it's own but F number depends
* how you package Lens module
*/
"2.8“, "4“, "5.6“, "8“, "11“, NULL
};
26
New V4L2 APIs for exposure control
New exposure metering control
+/* Exposure Methods */
+#define V4L2_CID_PHOTOMETRY (V4L2_CID_CAMERA_CLASS_BASE+17)
+enum v4l2_photometry_mode {
+
V4L2_PHOTOMETRY_MULTISEG = 0,
+
V4L2_PHOTOMETRY_CWA = 1,
+
V4L2_PHOTOMETRY_SPOT = 2,
+
V4L2_PHOTOMETRY_AFSPOT = 3,
+};
27
New V4L2 APIs for digital camera
Lens control
28
New V4L2 APIs for lens control
Need for enhanced Focus mode API
- Current V4L2 framework just slightly covers
MANUAL & AUTO FOCUS mode for usb webcam
- High-end camera modules support for various focus
mode presets like
• AF Macro
• AF-S / AF-C
For compatibility
Focus control API should be defined
29
New V4L2 APIs for lens control
Focus Mode & Enumeration values
#define V4L2_CID_FOCUS_ABSOLUTE
(V4L2_CID_CAMERA_CLASS_BASE+10)
#define V4L2_CID_FOCUS_RELATIVE
(V4L2_CID_CAMERA_CLASS_BASE+11)
+#define V4L2_CID_FOCUS_MODE (V4L2_CID_CAMERA_CLASS_BASE+13)
+/* Focus Methods */
+enum v4l2_focus_mode {
+
V4L2_FOCUS_AUTO
+
V4L2_FOCUS_MANUAL
+
V4L2_FOCUS_MACRO
+
V4L2_FOCUS_CONTINUOUS
+};
=
=
=
=
0,
1,
2,
3,
30
New V4L2 APIs for digital camera
Object recognition
31
What is object detection?
Object detection
Gee!
It recognized my face!
Hello everyone
32
What should we consider?
What do we set?
- Detect mode
- Type of object
- How many objects to detect
- Triggered action
What do we get?
- How many objects are detected
- Detected object’s type
- Detected object’s coordinates
33
New V4L2 APIs for object recognition
New API for object recognition
- New capability
#define V4L2_CAP_ASYNCIO
#define V4L2_CAP_STREAMING
0x02000000 /* async I/O */
0x04000000 /* streaming I/O ioctls */
+#define V4L2_CAP_OBJ_RECOGNITION
0x10000000
- New IOCTL
+#define VIDIOC_S_RECOGNITION _IOWR ('V', 85, struct v4l2_detect)
+#define VIDIOC_G_RECOGNITION _IOR ('V', 86, struct v4l2_detect)
34
New V4L2 APIs for object recognition
New API for object detection
+/* Object detection and triggered actions */
+enum v4l2_recog_mode {
+
V4L2_RECOGNITION_MODE_OFF
= 0,
+
V4L2_RECOGNITION_MODE_ON
= 1,
+
V4L2_RECOGNITION_MODE_LOCK
= 2,
+};
+
+enum v4l2_recog_action {
+
V4L2_RECOG_ACTION_NONE
= 0,/* only detection */
+
V4L2_RECOG_ACTION_BLINK
= 1,/* Capture on blinking */
+
V4L2_RECOG_ACTION_SMILE
= 2,/* Capture on smiling */
+};
+
+enum v4l2_recog_pattern {
+
V4L2_RECOG_PATTERN_FACE
= 0; /* Face */
+
V4L2_RECOG_PATTERN_HUMAN
= 1; /* Human */
+
V4L2_RECOG_PATTERN_CHAR
= 2; /* Character */
+}
35
New V4L2 APIs for object recognition
New API for object detection
+struct v4l2_recog_rect {
+
enum
v4l2_recog_pattern p;
/* What pattern detected */
+
struct
v4l2_rect o;
/* detected area */
+
__u32
reserved[4];
+}
+
+struct v4l2_recognition_data {
+
__u32
detect_cnt; /* detected object counter */
+
struct
v4l2_recog_rect obj; /* detected area */
+
__u32
reserved[4];
+};
+
+struct v4l2_recognition {
+
enum v4l2_recog_mode mode;
+
enum v4l2_recog_pattern pattern;
/* What pattern to detect */
+
__u32
obj_num; /* How many object to detect */
+
__u32
detect_idx;
/* select detected object */
+
struct v4l2_recog_data
data; /* read only */
+
enum v4l2_recognition_action action;
+
__u32
reserved[4];
+};
36
What should we consider?
What do we set?
- Detect mode
- How many objects to detect
- Triggered action
What do we get?
- How many objects are detected
- Detected object’s coordinates
37
New V4L2 APIs for object recognition
New API for object recognition
- New capability
#define V4L2_CAP_ASYNCIO
#define V4L2_CAP_STREAMING
0x02000000 /* async I/O */
0x04000000 /* streaming I/O ioctls */
+#define V4L2_CAP_OBJ_RECOGNITION
0x10000000
- New IOCTL
+#define VIDIOC_S_RECOGNITION _IOWR ('V', 85, struct v4l2_detect)
+#define VIDIOC_G_RECOGNITION _IOR ('V', 86, struct v4l2_detect)
38
New V4L2 APIs for object recognition
New API for object detection
+/* Object detection and triggered actions */
+enum v4l2_recognition_mode {
+
V4L2_RECOGNITION_MODE_OFF
= 0,
+
V4L2_RECOGNITION_MODE_ON
= 1,
+
V4L2_RECOGNITION_MODE_LOCK
= 2,
+};
+
+enum v4l2_recognition_action {
+
V4L2_RECOG_ACTION_NONE
= 0,/* only detection */
+
V4L2_RECOG_ACTION_BLINK
= 1,/* Capture on blinking */
+
V4L2_RECOG_ACTION_SMILE
= 2,/* Capture on smiling */
+};
+
+struct v4l2_recognition_data {
+
__u8 detect_cnt; /* detected object counter */
+
struct
v4l2_rect
o; /* detected area */
+};
+
+struct v4l2_recognition {
+
enum v4l2_recognition_mode
mode;
+
__u8
obj_num; /* How many object to detect */
+
__u8
detect_idx;
/* select detected object */
+
struct v4l2_recognition_data data; /* read only */
+
enum v4l2_recognition_action action;
+};
39
New V4L2 APIs for digital camera
Dual camera
40
Dual camera for 3G handset
What is dual camera?
Flip side camera
: for still shots
Front side camera
: for special purpose
Dual camera applications
- Video telephony
- Camera applications
• Taking self portrait shots
41
Dual camera for 3G handset
H/W restrictions in Application Processor’s camera peripheral interface
Mega
camera
VGA
camera
Data
path
42
Why should we make single device node?
H/W restrictions in Application Processor’s camera peripheral interface
“SINGLE” camera interface
on Application Processor !
Only one camera could be handled at a time
Only one camera could be using data pins at a time
Each camera is different in
Working MCLK
Working resolution
(SYNC)
Working PCLK
43
What do we have in V4L2?
We already have API for dual camera
- Enumerate video inputs
VIDIOC_ENUMINPUT
- Query or select the current video input
VIDIOC_G_INPUT
VIDIOC_S_INPUT
44
What do we need to consider for dual camera?
What do we need to consider?
•Let user space application get noticed about camera input devices
•Don’t forget VIDIOC_ENUMINPUT
•Check out supported pixel format using VIDIOC_ENUM_FMT and
VIDIOC_ENUM_FRAMESIZES
•Context of each camera module
•Capability : Supported color spaces, resolutions, additional functions…
•Latest working status : resolution, color space, effect and so on
•Context of camera interface
•Working information of the camera module getting switched : Clock
information and so on
45
Example of switching camera
How to switch active camera
Condition 1
How to enable camera module:
Give power source
Enable expecting camera
Give MCLK
Condition 2
Shared things
Camera power enable pin
Camera data pins
MCLK
Mega camera
VGA camera
Mega
Enable
VGA
Enable
Mega
Reset
VGA
Reset
CAM
POWER
MCLK
DATA PINS
CPU
Condition 3
Dedicated things
Camera enable pins
Camera reset pins
46
Abstract work flow of switching camera
Cam
power off
STREAMON
Camera (A)
initialize
Start
preview
Camera (A)
enable pin off
Camera (B)
Enable pin on
Cam
power on
Switch to camera
STREAMOFF
(B)
Stop
preview
G_INPUT/
S_INPUT
Camera (B)
initialize
Start
preview
Camera interface
reset & setting
Stop preview
Stop DMA
Reset camera
interface’s current
setting
Make a proper
MCLK for
camera (B)
Setup resolution,
color-space, DMA
47
Porting issues in SoC camera subsystem
48
I2C issue
I2C protocol has start and stop condition
Some camera modules generate unexpected noise until their power up
sequence
- Makes I2C bus in bus busy state
49
I2C issue
How to prevent this malfunctioning issue
Use a “noise free on power up” camera module
OR
Use a level shifter to block and ignore noise from camera module
OR
Change functionality of I2C pins while turning on camera module
50
Resolution issue
All
-
about between camera interface and camera module
Expected resolution from camera interface
Served resolution from camera module
Symptoms depend on camera interface
Expecting resolution is bigger than served resolution
- A “select time out” error in user space application or
- Preview image is floating in some direction
Serving resolution is bigger than expected resolution
- A “select time out” error in user space application or
- Preview angle looks like to be zoomed in
51
Color space issue
Color order mismatched between camera interface and camera module
52
Lens control issue
Preview freezing while checking lens response
- Using while() loop while it checks response, preview will freeze
until focusing job gets finished
- If the ISP driver use the kernel thread when it check response
from current lens status could make smooth preview on the
LCD while lens is on focusing.
53
Lens control issue
Solving preview freezing while checking lens status
struct platform_isp {
const struct platform_data *pdata;
struct v4l2_int_device *v4l2_int_device;
struct i2c_client *i2c_client;
int reschk;
. . . . . . . . .
}
static int ioctl_s_ctrl(struct v4l2_int_device *s,
struct v4l2_control *vc)
{
struct platform_isp *isp = s->priv;
struct i2c_client *client = isp->i2c_client;
switch (vc->id) {
case V4L2_CID_FOCUS_AUTO:
isp->vfm = vc->value;
if (vc->value == V4L2_CID_FOCUS_MACRO) {
err = i2c_command_for_isp( client, focus_macro );
isp->reschk = 1;
}
If isp->reschk is Set,
break;
The kernel thread is active.
}
}
54
Lens control issues
Solving preview freezing while checking lens status
int pseudo_reschk_thread (void *data) Platform Data Vairable for isp, or
static variable.
{
int err = 0;
This function return when the ISP is
………
matched the focus(+ return) or
do {
timeout(- return).
switch (isp->reschk) {
case 1:
err = i2c_command_for_isp(client, focus_check);
if (err < 0) {
isp->reschk = 1; /* i2c command function failed. */
} else {
isp->reschk = 0; /* if succeed, reschk ended. */
}
break;
……
This is able to re-check
}
to the next check time.
} while(1);
return 0;
}
55
JPEG data synchronization issues
JPEG has no synchronization protocol when it gets transferred
All about camera interface and camera module’s data synchronization
- When camera interface supports JPEG sync
• Follow user manual and make camera module to follow that protocol
- When camera interface does not support JPEG sync
• Make synchronization protocol: make JPEG data to be synchronized
with VSYNC
56
Q&A
57
Thank you
58